Techniques and workflows for computer graphics animation system

ABSTRACT

The disclosed implementations describe techniques and workflows for a computer graphics (CG) animation system. In some implementations, systems and methods are disclosed for representing scene composition and performing underlying computations within a unified generalized expression graph with cycles. Disclosed are natural mechanisms for level-of-detail control, adaptive caching, minimal re-compute, lazy evaluation, predictive computation and progressive refinement. The disclosed implementations provide real-time guarantees for minimum graphics frame rates and support automatic tradeoffs between rendering quality, accuracy and speed. The disclosed implementations also support new workflow paradigms, including layered animation and motion-path manipulation of articulated bodies.

RELATED APPLICATION

This application claims the benefit of priority from U.S. ProvisionalPatent Application No. 60/672,459, entitled “Generalized ExpressionGraphs With Temporal Caching For Use In An Animation System, And RelatedMethods,” filed Apr. 19, 2005, which provisional patent application isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The disclosed implementations are generally related to computergraphics.

BACKGROUND

Three-dimensional (3D) Computer Graphics (CG) animation systems are usedby a variety of industries (e.g., entertainment, advertising, etc.) togenerate animated content for movies, video games, commercials and thelike. Unfortunately, it can take a long time to produce animated contentdue to deficiencies associated with conventional 3D CG animationsystems. For example, it can take as long as four years to produce atwo-hour animated movie.

Traditional animators hold paper in hand while flipping between poses toget a sense of the motion before recording a pencil test. This gives theanimator an intuitive “feel” for the animation they are creating. CGanimators would like to flip back and forth between poses in the sameway, but the time it takes conventional 3D CG animation systems toupdate a new frame does not allow for instant visual feedback. Instantvisual feedback can speed-up the animation workflow, resulting in lesstime to generate the final product.

The tools of a traditional animator are simple: a pencil and eraser.These tools are easily mastered so that the tools become an extension ofthe mind of the artist. Conventional 3D CG animation systems are morecomplex to master. Ideas can take a winding path from the mind of theartist, through the mouse or stylus, then through the software userinterface, and finally to the character rig itself. Conventional 3D CGanimation systems often provide user interfaces that are too complex orthat are not intuitive, resulting in the animator spending more timelearning the animation system than on animating.

Animation is an experimental process. Even veterans of the art form liketo try something new without worrying that it will destroy the work theyhave already completed. In traditional animation, drawings are easilyerased and drawn over. Conventional CG animation systems often do notallow non-destructive experimentation, forcing the animator to makecompromises on creativity to meet deadlines.

Traditionally, animation is a team effort. Communication and artworkflow back and forth between departments, animators, assistants,supervisors, directors, managers, and even studios collaborating atgreat distances. Conventional CG animation systems often fail to providefeatures that facilitate collaboration among team members, resulting inunnecessary delays in the animation process.

SUMMARY

The disclosed implementations describe techniques and workflows for a CGanimation system. In some implementations, systems and methods aredisclosed for representing scene composition and performing underlyingcomputations within a unified generalized expression graph with cycles.Disclosed are natural mechanisms for level-of-detail control, adaptivecaching, minimal re-compute, lazy evaluation, predictive computation andprogressive refinement. The disclosed implementations provide real-timeguarantees for minimum graphics frame rates and support automatictradeoffs between rendering quality, accuracy and speed. The disclosedimplementations also support new workflow paradigms, including layeredanimation and motion-path manipulation of articulated bodies.

In some implementations, an animation method includes: providing adisplay environment for presentation on a display device; providing abody for display in the display environment; and determining a motionpath for a first portion of the body relative to a second portion of thebody.

In some implementations, an animation includes: providing a displayenvironment for presentation on a display device; providing a body fordisplay in the display environment; providing a first motion path for afirst portion of the body relative to a second portion of the body;providing a second motion path for the first portion of the bodyrelative to a second portion of the body; and providing a mechanism foranimating the first portion of the body to follow either the first orsecond motion path.

In some implementations, a method of creating expression graphs for ananimation system includes: providing a graphical user interface;receiving a first node selection from a plurality of node types;receiving a second node selection from the plurality of node types;displaying graphical representations of the selected nodes in thegraphical user interface; and providing a connection mechanism forenabling a user to connect at least one output of the first node with atleast one input of the second node to form a graphical structure thatrepresents an expression.

In some implementations, a method of creating an expression graph for ananimation system includes: providing a text-based programming languagefor specifying expression graphs for an animation system; executing aprogram developed with the programming language to generate anexpression graph; and animating a body in a display environment usingthe expression graph.

In some implementations, an animation method includes: providing adisplay environment for presentation on a display device; providing afirst body for display in the display environment; providing a secondbody for display in the display environment; and determining a motionpath for a portion of the first body relative to the second body.

In some implementations, an animation method includes: providing adisplay environment for presentation on a display device; providing abody for display in the display environment; and determining a motionpath for a portion of the body relative to a coordinate systemassociated with the display environment.

In some implementations, an animation method includes: providing adisplay environment for presentation on a display device; providing abody for display in the display environment; evaluating an expressiongraph associated with the body at a first rate; processing body geometryat a second rate; and rendering the body in the display environment at athird rate, where the first and second rates are decoupled from thethird rate.

Other implementations are disclosed that are directed to systems,methods, apparatuses and computer-readable mediums.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an exemplary data flow network.

FIG. 2 illustrates an exemplary inverse-kinematics solver node.

FIGS. 3 a and 3 b are examples of hierarchy inversion via subtreere-rooting.

FIG. 4 is a screenshot of exemplary graphical user interface forcreating, viewing and editing graph structure.

FIG. 5 illustrates the interconnection of nodes using the graphical userinterface shown in FIG. 4.

FIG. 6 illustrates a pull-out drawer for changing internal nodesettings.

FIGS. 7 a-7 c are screenshots of exemplary manipulators for controllingdilation and tickertape.

FIGS. 8 a-8 c are screenshots illustrating a bendable-bottle model.

FIGS. 9 a-9 c are screenshots illustrating a character walking in place.

FIG. 10 a is a screenshot illustrating the user selection of a rootjoint of the character in FIGS. 9 a-9 c with tickertaping enabled.

FIG. 10 b is a screenshot illustrating a lateral shift of the motionpath shown in FIG. 10 a, so that frame 9 on the path lies at the rootjoint.

FIG. 11 is a flow diagram of an exemplary motion path manipulationprocess.

FIG. 12 is a block diagram of exemplary user system architecture.

DETAILED DESCRIPTION

1.0 Generalized Expression Graphs With Temporal Caching

It is common practice in computer graphics to represent the geometry ina scene with directed acyclic graphs (DAG's) while possibly maintaininga separate expression graph for update of the scene geometry. Wedescribe a method of representing scene composition and underlyingcomputations within a unified generalized expression graph with cycles.Our system provides natural mechanisms for level-of-detail control,adaptive caching, minimal recompute, lazy evaluation, predictivecomputation and progressive refinement. It provides real-time guaranteesfor minimum graphics frame rates and supports automatic tradeoffsbetween rendering quality, accuracy and speed. It supports new workparadigms described herein, including layered animation and motion-pathmanipulation of articulated bodies.

1.1 Terminology

At the core of our system lies an expression graph that generalizes thescene tree data structure traditional to computer graphics. The terms“graph” and “tree” here come from the mathematical field of graphtheory. A graph is a set of nodes and a set of edges, or lines, thatinterconnect the nodes. The edges connecting to a given node areincident to that node. If, starting from one node A, you can follow aseries of incident edges and get to a second node B, then there is apath between A and B; the number of edges you traversed in getting fromA to B is the path length. If there is a path from each node in a graphto every other node, then the graph is connected. If there is anon-empty (length>0) path from some node A back to A, then this loop iscalled a cycle and the graph is cyclic. A graph with no cycles isacyclic.

Throughout the discussion that follows, when referring to graphs we willmean directed graphs or digraphs, that is, graphs in which the edgeshave direction. Typically when drawing a digraph we show the nodes ascircles or rectangles, and the edges as arrows between nodes. If anincident directed edge points to node A, it is an incoming edge of A. Ifit points away from A, it is an outgoing edge. If an edge e is from A toB, we say A is the origin of e, and B is the destination of e.

Digraphs may be used to denote mathematical expressions. For example,the expression “(3+4)*6” may be represented as a digraph in which “3”,“+”, “4”, “*”, “6”, are denoted by nodes, and the steps of combiningthese elements are edges between the nodes. Thus there are edges from“3” to “+” and from “4” to “+”; and from “+”, to “*” and from “6” to“*”. Such an expression graph provides a compact representation allowingevaluation of the mathematical expression. Moreover, the graphillustrates dependencies in the expression; for example, in theexpression above, that the addition occurs before the multiplication,i.e., that the result of the multiplication depends on the addition, butnot vice versa.

A tree is an acyclic, connected graph. We can designate a unique node ofa tree to be the root; then the tree is a rooted tree, and there is animplicit direction given to all edges in the tree pointing away from theroot. All nodes in a rooted tree have zero or one incoming edges andzero or more outgoing edges. Only the root node has zero incoming edges.Nodes with zero outgoing edges are leaves.

Trees are traditionally used in computer graphics to represent thecontents of a 3-dimensional (3-D) scene. Individual nodes in the treemay denote objects that are rendered to the screen, for example,geometry (e.g., a group of polygons describing the surface of anobject), or geometric transformations (e.g., a rotation), lightsproviding virtual illumination of the scene, virtual cameras, etc. Sucha structure is referred to as a scene tree.

1.2 Generalized Scene Graph

As is common in graphics systems, we use a graph for representingdependencies in a 3-D scene. We describe a construction on graphs usedfor representing our 3-D scene data and the computational dependenciesthat underlie that data. In contrast with traditional scene trees, ourgraph is a generalized directed graph that may contain cycles. Thisgraph represents not only our scene geometry in the way a scene treedoes, but further represents the hierarchy of computations that governthe motion and interrelationships between scene elements. Our expressiongraph is sufficiently general to describe any computational process thatcould be described in a general-purpose programming language;specifically it is Turing complete. Indeed, we describe a text-basedcomputer language in Section 2 for use in specifying these graphs thatis capable of expressing any algorithm or process that could bedescribed in a programming language such as C.

Within this expression graph, we retain the notion of embedded subgraphsthat correspond to traditional acyclic, rooted scene trees. However,there is not a single such grouping that constitutes a unique scene treein our graph. Rather, the same graph may have many overlapping groupingsthat each constitutes a scene tree, and these groupings may change overtime. We may dynamically enable and disable edges, change edge arrowdirections, and partition the graph into subgraphs each with its ownscene tree. Though it is often convenient to designate a root node, thisis not necessary, and in fact at times we may designate a different nodeof a particular subgraph to be the root of that subgraph. Embedded scenetrees and the dynamic nature of graph topology are described in detailin Sections 1.9 and 1.17.

1.3 Data Flow Network

FIG. 1 illustrates an exemplary data flow network 100. Such a networkrepresents a computation by encoding data 102 (e.g., a number) oroperations 104 (e.g., addition) within nodes, while edges 106 denote thedependencies between nodes. We think of data as flowing along the edgesin a graph in the direction the edge arrows point. As is common in suchnetworks, nodes may internally maintain a cache of the value theyrepresent within the total expression. We maintain in each node a stateflag, or dirty flag, that indicates whether the data cached at that nodeis currently valid. We refer to data cached in a dirty node as staledata.

The graph implements a push-pull network, which models the statedependencies of computational objects underlying a character or a scene.Nodes “pull” against the direction of the edge arrows to get needed datafrom other nodes higher “up” in the graph, and when data changes nodespush an indication of the change (though not necessarily the dataitself) “down”, in the arrow direction, to nodes that are dependent onthat data. This way, if a node A has pulled data from another node B, Aknows it does not need to pull the data again until it receives a pushfrom B.

Because our graphs may be cyclic, the terms “up” and “down” are usedloosely. In fact, a node may be “down” (or “up”) from itself, in theevent it is part of a directed cycle, that is, if while following fromthe node a series of edges in (or against) the direction of the edgearrows one will encounter that node again. For convenience, we will meanthe directions “down” and “up” to refer to traversing edges in thedirection of and against the direction of edge arrows, respectively.

Initially all nodes in the graph are dirty (i.e., their dirty flag isset). When an output value at a particular node is needed, we pull onthe node. If the node is clean (i.e., the dirty flag is cleared), thevalue stored at that node is returned. If the node is dirty, the nodepulls on its incoming edges (its inputs). Pulling on an edge fetches thevalue of the origin node of the edge by pulling on that node. Such apull may recursively propagate upwards through the graph. In this way,the subgraph upstream of a node is pulled clean. Now the nodere-evaluates itself, stores a new value in its cache, marks itself asclean, and returns the value to the node that initiated the pull.

Conversely, an external process may cause the value at a node to change.Now the values stored at nodes downstream are marked dirty. The nodepushes its output edges, which in turn push their destination nodes,which in turn push their outputs, and so forth. In this way, thesubgraph downstream of a node is pushed dirty.

This push-dirty, pull-clean mechanism allows us to evaluate sub-piecesof the expression graph while doing a minimal recompute. That is, wepreferably do not recompute more than is necessary based on what data wehave already computed and cached within the graph and what has changedto invalidate our prior computations. Significant in this scheme is thatwhen something changes, we don't immediately recompute anything, wesimply note the scope of the effect (by pushing dirty through theaffected subgraph). We will later pull-clean subgraphs only as needed.This in effect provides lazy evaluation of the graph.

1.4 Cyclic, Multi-Cached Graph Evaluation

Our graphs differ from traditional data flow networks in many respects.Our graphs may be cyclic, that is, they may contain loops. Normally dataflow networks are acyclic, because cycles in such a network could leadto an infinite cycle of pushes or pulls and thus a computer crash. Weemploy a marker mechanism, described below, to provide controlledlooping or recursion through such cyclic structures.

The data flowing along edges in our system conceptually are

parameter,value

pairs, where often (but not always) the parameter refers to time, andthe value designates the state of a node output at that time. Thus,pulls upwards in the system are for data at a particular parametervalue, while pushes downward signaling state invalidity will indicatethe parameter intervals that the state change affects. That is, Pull(t)yields a pair

t,v

, and invalidation messages take the form Push(Γ), where Γ=(σ₁,σ₂,σ₃, .. . ) gives the closed intervals σ_(i)=[t_(a),t_(b)]={u|t_(a)<=u<=t_(b)}over which invalidation is to occur. A pulled node itself performs aPull(t) on its inputs if necessary (and so on, recursively upwardsthrough the graph). A node may use this t internally in the calculationit performs on its inputs in producing its output values, or it maysimply pass it upwards via the pull. A node may even perform a pull at adifferent t than it itself was pulled at, or it may perform multiplepulls at different t's.

It is up to each individual node whether or not to retain internally thedata only for the most recently requested parameter value, or to cachedata across many parameter values. We refer to the latter as “temporalcaching”, since usually it is used to cache data over a range of times.The decision whether or not to employ temporal caching within aparticular node may be made automatically, as described below, or it maybe set by a human operator.

When a node pulls for data, it may either designate an explicitparameter value for which it wants the data, or alternatively it may askfor data at an unspecified “input” parameter value (designated herein askInputParam). This second case is only allowed when the source of thedata depends on a parameter input from a “parameter driver” node,typically a clock, somewhere up the graph. Pulling on a parameter driveryields the “current” value for that parameter. In this second case then,the pull at kInputParam is implicitly a request for the data as it wouldbe at the “current” parameter values for all drivers upwards in the pullchain. In the simple case of a single parameter driver that is theanimation clock, pulling at kInputParam yields the value of a nodeoutput for the current animation time. Pulling at another parametervalue t yields the node output data for time t. The kInputParam token isdiscussed further in Section 1.18.1.

Because the graph may contain cycles, we take care not to pull endlesslyalong a series of edges that lie in a cycle. Unchecked, such a cyclicpull would lead to an “infinite loop” or “infinite recursion” and aprogram crash. We guard against this by raising a marker flag in eachnode as it begins its pull, and lower the flag once it has received itsdata. If a node receives a pull while its marker flag is raised, itknows the pull is recursive. It then has a choice to either signal anerror, to return an estimated, default, or stale value; or to allow somefinite number of recursions to occur by keeping count of how many pullsit receives while its flag is raised. The human operator who assemblesthe graph makes the choice among these options. We use the last option,of allowing some finite level of recursion to occur, to build subgraphsthat are capable of recursive computations, such as iterativeerror-minimization algorithms for solving inverse kinematics anddynamics problems. We use a similar technique to provide progressiverefinement of expensive algorithms as described in a later section.

When a node pushes a state invalidation message r, this messagespecifies the (closed, possibly disjoint) interval of parameter valuesover which the invalidation has occurred. Again considering the usualcase where the parameter refers to animation time, the push designatesthat the output of the node is invalid over some set of animationframes. The nodes that receive this push (that is, the nodes to whichedges point from the node that initiated the push), can themselves pushalong their output edges, and this continues recursively. In this way,the graph downward from the initiating node is flooded withstate-invalidation messages. Because the graph may contain cycles, wetake care not to continue this recursion through a cycle, as this wouldlead to an “infinite loop” or “infinite recursion” and a program crash.We guard against this simply by not pushing dirty intervals past nodesthat are already dirty over at least the interval being pushed. Byinduction, we know that the entire subgraph below such a node is alreadyalso dirty over at least that interval, so there is no point incontinuing the push.

1.5 Adaptive Caching

An individual node may contain no cache of its current value, maycontain a single-line cache of the value for the most recently pulledparameter t, or may contain a multi-line cache of values for varioust's. In the case where t denotes time, we refer to multi-line caching astemporal caching. If a node contains no cache, then effectively it isdirty: any pull on it will result in it pulling upstream. If a nodecaches a single value, computed for some parameter t, then it is cleanat t but dirty at all u≠ t. If a node caches a set of values, then ithas a set of dirty bits, one per cache line, and it will be clean oversome (possibly empty) set of disjoint intervals. As detailed below,these cache lines are not infinitely sharp, that is, a parameterinterval around t will be mapped to t's cache line. We say such cachelines have a non-zero cache line width.

Nodes may have more than one output, and each output will have aseparate cache. Outputs of the same node that use the same cachingscheme will share a common set of dirty flags. In some cases, a node mayhave an output, but no outgoing edge is connected to that output. Inthis case, it is as if the node did not have the output, and no value iscomputed for or cached in that output.

Node outputs themselves decide which caching scheme (none, single, ormulti-line) they will use, or the user may explicitly designate whichscheme a particular output shall use. The scheme used at a particularoutput may change dynamically during program execution, eitherautomatically or because the user changes it.

Automatic selection of a caching scheme involves a cost-benefitcalculation. Cost of a caching scheme is the cost of the data copyduring cache write plus the memory storage cost. Using no caching has nocost; single-line caching costs a copy on cache-write for each cachemiss plus memory needed for storing one cache slot; multi-line cachingcosts a copy on cache-write for each cache miss plus memory for storingn cache slots. Thus, we calculate a unitless, abstract cost of cachingan output as: ${cache\_ cost} = \{ \begin{matrix}{k*{cache\_ slot}{\_ size}*( {1 + n} )} & {{when}\quad{cached}} \\0 & {{when}\quad{uncached}}\end{matrix} $where cache_slot_size is the size of the value type for the output, andthe “1+” reflects the cost of the cache write for cached outputs.

The benefit of a cache scheme depends on how often the node is pulled,how effective is its cache, how expensive are cache misses, and howimportant is the node. The pull rate on a node output V is the number oftimes V is pulled per unit time. The cache hit ratio of V is the numberof cache hits for V divided by the total number of pulls on V (windowedover a unit time interval); for uncached outputs, the cache hit ratiowill be zero. The cost of a cache miss is the time spent on average inre-evaluating the node following a miss; when a node or node output hashigh evaluation cost, we say that node or output is expensive. Theimportance of a node is given by that node's priority, as described infollowing sections. (As detailed elsewhere, the priority reflectssalience due to projected screen area and user focus.) Expensive outputsof high-priority nodes with a high pull rate but a low cache-hit ratioare good candidates for a more aggressive caching scheme. Finally, wedefine the periodic evaluation cost of a node as the aggregate timespent re-evaluating that node over some sliding-interval window. Thus,cache_benefit=evaluation_cost*priority*pull_rate*cache_hit_ratiocache_inadequacy=evaluation_cost*priority*pull_rate*(1−cache_hit_ratio),and we track these values (as sliding-window averages) for every outputwhose cache-selection method is “automatic” and that belongs to a nodewhose periodic evaluation cost exceeds some fixed threshold.Periodically (following frequent cache misses or occasionally duringidle time) we reconsider the caching scheme used at such an output. Weevaluate:cache_worth=cache_benefit/(1+cache_cost)using the cache cost of the current caching scheme, andcache_need=cache_inadequacy/(1+cache_cost),using the cache cost of the next most aggressive caching scheme.

Outputs with the highest cache_need are switched to a more aggressivecaching scheme. Outputs with the lowest cache_worth and lacking a highcache_need (or that are already using multi-line caching) are switchedto a less aggressive caching scheme. Also, outputs which are regularlydifferentiated or integrated (Section 1.14), or that drive a motion path(Section 4.1) automatically use multi-line caches when feasible.

For nodes employing multi-line caches, the mapping from parameter t to aline in the cache is determined by a cache-mapping function M. Thisfunction depends on the total parameter range [t_(min), t_(max)] that iscached, as well as the number of cache lines, and the width of thoselines. We use the cache mapping function:s=M _(c) ₀ _(c) _(r) (t)=c ₀+round(c _(r) t)

and say t maps to cache line s under M if and only if(0≦s<N)ˆ(abs(c_(r)t—round(c_(r)t))≦c_(tol))

where:

-   -   t=the parameter value at which lookup occurs,    -   s=the cache line to which t maps,    -   N=the number of lines in the cache (cache lines are numbered        from 0 to N-1),    -   c₀=the cache line corresponding to t=0,    -   c_(r)=the number of cache lines per unit change in parameter t,    -   c_(tol)=the cache tolerance, equals ½ the cache line width,    -   round(x)=x rounded to the nearest integer,    -   abs(x)=the absolute value of x, that is,    -   abs(x)=x for x≧0, otherwise abs(x)=−x, and ˆ=logical AND, that        is,    -   aˆb is true if and only if both a is true and b is true.

If t in [t_(min), t_(max)] maps to some cache line s and s is markedclean, we say a cache hit occurred, otherwise we say there was a cachemiss. We define the cache inclusion ratio as:C_(inclusion)=2c _(tol)c_(r).

If c_(inclusion)=1, all parameter values t within [t_(min), t_(max)]yield valid cache mappings; the cache slots are maximally wide (theyabut one another). This provides us with potentially inaccurate but veryfast graph evaluation, since all t in this range will map onto thecache. By allowing c_(inclusion)<1, we increase the accuracy of graphevaluation at the expense of evaluation speed, since as c_(inclusion)decreases, fewer queries will result in cache mappings, but the t thatdo map onto the cache will be better centered within the cache lines. Atc_(inclusion)=0, only exact cache mappings are allowed; the cache slotsare infinitely thin. Varying c_(inclusion) in the range [0,1] providesus with a continuous Level of Detail (LOD) control on graph evaluation,enabling us to vary the trade off between speed and accuracy. We cancontrol this tradeoff per-node by allowing each node to specify its ownc_(inclusion). This is just one of several LOD mechanisms available, asdescribed next.

1.6 Approximate and Partial Evaluation

When a cache hit occurs in response to a query Pull(t), the noderetrieves the value v stored in cache line s, and returns the tuple

t,v

. If a cache miss occurs, the node may do any one of:

-   -   1. return the value v_(stale), which is the value stored at the        nearest cache line to s, even if that cache line is marked        dirty;    -   2. return the value v_(approximate), interpolated between the        nearest neighboring cache lines s⁻ and s⁺ that are non-dirty,        where s⁻<s and s⁺>s;    -   3. Pull(u) on each of its incoming edges, where normally u=t,        then recompute its output values v_(i) for each output i.    -   4. Pull(u) on each incoming edge, but only partially recompute        its output values.

When (3) occurs, we say the node re-evaluates. If t maps to some cacheline s, then the node will place each v_(i) in that cache line foroutput i and mark the cache flag for line s as clean.

When (4) occurs, we say the node does a partial re-evaluation. The nodewill not store any value in the output caches and will not change thestate of any cache flag. It will store enough information to enablelater continuing evaluation where it left off. In returning

t,v

to the node that initiated the Pull(t), it will include a special tokenthat says this is a partial or intermediate result and needs furthersubsequent refinement. This provides a mechanism for progressiverefinement of complex computations within the graph. Further, itprovides the ability to create graph structures supporting looping andrecursion.

Nodes automatically choose between mechanisms (1)-(4) above based on theurgency and level of detail of the pull. The urgency is a global value,computed by the application, that reflects how much time is left beforethe total graph evaluation is complete so that the updated 3-D scene maybe drawn on the screen. (Urgency and level-of-detail are described inSection 1.7.) Ideally, all cache misses result in (3) occurring, thatis, dirty nodes are fully re-evaluated and dirty subgraphs are pulledclean. In some circumstances, we may begin running out of time whiledoing this graph re-evaluation; the application will raise the urgencyof the evaluation as the deadline for drawing the next frame approaches.This will result in dirty nodes employing strategies (1), (2), or (4),as detailed in Sections 1.7 and 1.8 below. Preference between strategies(1) and (2) for a given node is hard-coded into each node type and maybe overridden by the user or model builder per-node. Generally strategy(1) is preferred over strategy (2) because it involves no extracomputation and usually it provides the least-surprising behavior.Strategy (1) tends to create a delay or “lag” during direct manipulationwhen very expensive (or very low-priority, e.g., due to small screensize) sections of the 3-D scene don't update at the full frame rate.

This flexibility in providing stale, approximate, or partial resultsprovides us with real-time guarantees on the execution speed of thegraph evaluation, and ensures we can maintain a target frame rate inredrawing the scene. This provides still another LOD-mechanism, tradingoff accuracy and/or node recompute rate in order to maintain frame rate.More generally, it decouples the frame rendering rate from nodeevaluation rates, and even the evaluation rates of different nodeswithin the same graph. We further generalize this level-of-detailcontrol within the Pull( ) mechanism, as described below underLevel-of-Detail (Section 1.7). The usage of this LOD control to achieveconstant frame-rate is described below under Real-Time Guarantees(Section 1.8).

1.7 Pull Level-of-Detail

We now extend the Pull(t) semantics to include a specification of thelevel of detail (LOD) for which the value is requested. We sayPull(t,lod) yields

t,v

. (The lod is actually a vector quantity, but we discuss it first as ascalar quantity for simplicity.) This level-of-detail control canprovide a switch between multiple alternative representations, forexample, geometry may be multiply represented within the graph with thelod-mechanism used to automatically select the best representation. Thelevel-of-detail can also be used internally within nodes to selectalternative algorithms or to set the desired accuracy of an algorithm,for example, by controlling the refinement level of a subdivisionsurface.

The initial lod value originates at the application level and is passedthrough a sequence of Pull's or Render's. (Render calls are described inSection 1.9 below.) Nodes may modify lod as they pass it along for theirown internal reasons. For example, a node may want to evaluate an inputsubgraph very roughly as one step of its own internal evaluation, so itmay initiate the Pull on that input at a low lod. The user mayexplicitly raise or lower the relative lod of particular nodes through adialog in the user interface or programmatically during modelconstruction. As described in the next section, the application canautomatically lower lod globally as needed between or even during renderpasses to attempt to provide lighter-weight, and therefore faster, graphtraversal. This allows the application to trade-off rendering detail andaccuracy with speed to maximize quality while meeting frame-rate goals.

Interpretation of specific numerical values of lod is left to the nodesthemselves; at the graph level, lod is regarded as an abstract, unitlessvalue that is normally greater than or equal to zero. Zero lod specifiesthat the simplest/fastest possible representation or algorithm should beused. Lod greater than zero specifies that a more-accuraterepresentation should be used. A negative lod specifies that noevaluation or rendering should be performed at all; what happens insteaddiffers between Pull's and Render's. A Pull at negative lod will resultin cached, stale, or incomplete values being returned, as described inSection 1.6. The connection between negative lod and increasing urgencyis detailed below in Section 1.8.1.

For a Render with negative lod, neither that node nor its children willbe rendered at all. For Render's at zero lod, if an imposter (a polygontexture-mapped with an image of an object saved from a previous render)is available, the imposter will be drawn in place of the object itself;otherwise some extremely lightweight rendering (such as a bounding box)is used.

While we have discussed lod as if it were a single scalar (real) value,in fact we maintain distinct levels-of-detail for geometry,articulation, deformation, shading and lighting. That is, rather thanmaintaining and passing a scalar lod, we rely on a vector-valuedlod=(lod_(geom), lod_(artic), lod_(deform), lod_(shade), lod_(light)).Particular types of nodes generally will depend only on a single one ofthese components. The lod component(s) to which a given node typeresponds are referred to as that node type's LOD control channel(s). Forexample, mesh deformer nodes generally respond just to lod_(deform),while nodes representing light-emitting surfaces may respond to bothlod_(geom) and lod_(light). Again, it is left to each node to decide howto interpret the values within lod, except that all node types returnfrom Pull and Render calls immediately when any one of their controlchannels fall negative.

1.8 Real-Time Guarantees

We desire to provide soft-real-time guarantees on the maximum time takento traverse the graph for a given operation (typically renders). Weachieve this through various mechanisms:

-   -   dynamically adapt LOD    -   partial evaluation    -   progressive refinement    -   adaptive caching    -   parallel evaluation    -   predictive precomputation        1.8.1 Dynamic Level-of-Detail

As mentioned in the previous section, the system can dynamically adaptlevel-of-detail at the application level between and during rendertraversals. After each render traversal, the application notes how muchtime was taken relative to the target traversal rate. If the traversalwas considerably faster than necessary, the application increases thetop-level lod. The next render pass will be initiated with this higherlod. Conversely, if the traversal took longer than desired, theapplication will reduce the top-level lod.

The application can adapt lod globally during the course of recursivePull's or Render's by adjusting a global urgency value. Urgency isinitially 0.0 when the application itself initiates a Pull or Render. Astime passes, a watchdog thread will increase urgency towards 1.0 as thetime approaches the traversal time deadline. In the event the deadlineis exceeded, urgency rises above 1.0. Each call to Pull and Rendermultiplies its lod parameters by 1.0—urgency and uses the results as itseffective lod's. Thus, effective lod's will fall as the deadlineapproaches, encouraging the use of simpler and faster representations.If the deadline is exceeded, effective lod's will become negative,preventing any further evaluation or rendering and ensuring anear-immediate return from traversal recursion.

In this way, based on how well a given render pass has met its real-timedeadlines, the application will adjust its own top-level lod, attemptingto maximize quality and accuracy of the rendering without exceedingdeadlines. When the application is too ambitious, using an excessivelyhigh lod, it will raise urgency to ensure nonetheless that traversalterminates in soft real-time.

1.8.2 Partial Evaluation

As described in Case 4 of Section 1.6, there are circumstances in whicha node may return a pulled output value before it has completedre-evaluation. This happens when a node begins re-evaluation (i.e., apull occurs) with urgency <1.0, but urgency rises above 1.0 (i.e.,effective lod becomes negative) during evaluation. It may also happenbecause the time spent in a single invocation of the node evaluationfunction exceeds an application-specified time limit. Nodes may alsoelectively prematurely terminate a particular evaluation, for example,because a particular algorithmic condition occurs within the evaluationfunction (such as exceeding some set number of iterations within aloop). In all cases, graph traversal proceeds normally following thereturn from the partially evaluated output, except that the outputsubgraph of that output continues to be marked dirty. Subsequent Pull'son that subgraph will in turn pull on the partially re-evaluated node,eventually causing it to fully re-evaluate, at which point the outputsubgraph will be marked clean, as described previously.

Similarly, as identified in Cases 1 and 2 of Section 1.6, a pulled nodewill return stale or approximate results when the effective lod of thePull is negative (i.e., when urgency >1.0). Again, the subgraph belowthe node will remain dirty, and subsequent Pull's on the subgraph willcontinue to pull on that node. Due to other sections of the graphbecoming clean and/or cached, these subsequent pulls should occur withgreater time available, so that effective lod is eventually non-negative(i.e., urgency <=1.0) upon reaching this node. Recall that, followingSection 1.5, caching strategies within the graph will be automaticallyadjusted during these repeated pulls, shifting resources as necessary toallow us to Pull with non-negative lod. One circumstance in which we maynot be able to achieve a non-negative-lod Pull at a given node is if themere process of traversing edges upwards in the graph to reach that nodeexceeded the available time bounds for graph traversal. Because edgetraversal itself is an extremely lightweight (fast) operation, only anenormously large graph would exhibit this problem. Clearly such a largegraph would be beyond our ability to recompute effectively, and we doubtsuch graphs will be encountered in practical situations. However, thisdoes highlight that the maximum path length in a given graph givesimportant information about how costly evaluation of that graph may be.

The native ability of the graph to partially or approximately recomputeprovides a mechanism for achieving rough results when real-time demandsdo not allow for a full recompute.

1.8.3 Progressive Refinement

If we perform partial evaluation over multiple successive traversals, wewill progressively refine pass accuracy. In the case of elective partialevaluation, we can compute an error metric and bound, returning cleanwhen we have achieved the error threshold. Similar to what occurs withstrategy (1) in Section 1.6, this tends to create a delay or “lag”during direct manipulation when very expensive (or very low-priority,e.g., due to small screen size) sections of the 3-D scene don't updateat the full frame rate. In the case of progressive refinement, thisgives the effect of computationally heavyweight scene componentsgradually updating in response to rapid user direct-manipulations. Forexample, a complex deforming surface may change shape gradually overseveral frames despite the fact that the underlying skeleton is redrawnat an interactive frame rate.

1.8.4 Adaptive Caching

As mentioned above and in Section 1.5, the caching strategy within agiven node may change over time in response to observed efficacy of anyexisting cache and cost and frequency of node re-evaluation. Thisprocess is integral to our ability to sensibly manage resources toenable high-priority (e.g., large screen area or user-focused) subgraphsto re-evaluate rapidly. Thus, adaptive caching is an important enablerof the real-time evaluation methods described in this section.

1.8.5 Parallel Evaluation

Given that we can traverse a graph node-by-node, pulling input subgraphsclean or rendering output subgraphs, we may wish to break up traversalsof subgraphs into separate threads. This enables symmetricmultiprocessing on shared-memory architectures. Every recursive pull orrender presents the opportunity to spawn a new thread. We trackpull/render cost (as described for evaluation cost in Section 1.5 andelsewhere) and use that as a guide of when to spawn a new thread. Theideal situation is a node in which two expensive Pull's or Render's areperformed on large disjoint subgraphs. In this case, ideally we traverseboth subgraphs simultaneously. In fact, this occurs quite commonly inthe form of Render's at the scene graph root: different large top-levelmodels ideally will be evaluated on different processors. Potentiallythis provides an order-n speedup when running on a machine with nsymmetric processors.

1.8.6 Predictive Precomputation

It is common wisdom that the vast majority of processor time in thetypical computer is spent idle waiting for user actions. We can takeadvantage of idle time to re-evaluate dirty nodes. This is particularlyvaluable in conjunction with temporal caching: we re-evaluate nodes attimes corresponding to dirty cache slots, thereby filling those cacheslots with clean values. We refer to this as predictive evaluation,because we are anticipating that the outputs of these nodes will bepulled at parameter values other than the current value, although nosuch pull has yet occurred.

We prioritize nodes for predictive re-evaluation based on the product ofpriority and cost, with those measures computed as described in Section1.5. The greater is the product, the higher is the priority forpredictive re-evaluation. If we have an estimate of the error in a givencache slot (this may be easily computed for certain node types, such asthose performing subdivision or similar refinement algorithms), we mayuse that error estimate to scale the re-evaluation priority. Similarlywe may track how much time has passed since a cache-slot was lastre-evaluated, and scale priority by cache-slot age. Scaling priority byerror is useful for nodes that employ Case 2 of Section 1.6 (returnapproximate value). Scaling priority by age is useful for nodes thatemploy Case 1 of Section 1.6 (return stale value).

At the application level we maintain a priority queue that ranks nodesaccording to: priority * cost * age for nodes that have previouslyreturned v_(stale), priority * cost * error for nodes that havepreviously returned v_(approximate) and possessing an easily observableerror metric, or priority * cost for nodes that are not Pull-ed withnegative effective lod (or that lack any simple error metric).

For efficiency, we only bother to include in the priority queue thosenodes that have both a high priority and a historically high actualcost. Other nodes we expect to be poor candidates for predictivere-evaluation. In practice, this means we include in the priority queuethose nodes for which temporal caching is enabled on one or moreoutputs.

At program startup, we create one (or n, on n-processor systems)low-priority idle thread(s). When the system is busy, the idle threadswill sleep, yielding CPU time to more-urgent work. When idle timebecomes available, an idle thread will run, pulling a node forre-evaluation from the top of the priority queue. We evaluate the nodeat those times at which its cache slots are dirty. This evaluation mayresult in Pull's on other nodes; if these nodes have temporally cachedoutputs and no other thread is re-evaluating them yet, this idle threadwill also pull these ancestor nodes from the priority queue.

In this way we wander the graph filling caches during idle time, helpingto ensure that we will be able to respond rapidly to further useractions, especially frame changes, which tend to be the most-challengingaction to consistently perform in real time. We refer to this process aspre-caching, since we are filling caches in advance of when the cacheddata is needed. This mechanism is especially effective in conjunctionwith symmetric multiprocessing, allowing us simultaneously to handleuser interaction and to prepare for anticipated future demands on graphevaluation.

1.9 Render

The ultimate goal of graph traversal generally is to draw something onscreen. As with prior systems, we accomplish this via a top-down rendertraversal, distinct from the bottom-up evaluation traversal initiated bya Pull. The render traversal occurs over a connected subgraph of specialexpression nodes called scene graph nodes (or simply scene nodes); werefer to this subgraph as the scene graph. All scene nodes possess actm_in input and a ctm_out output (and possibly additional node-specificinputs and outputs). The parent of a scene node A is the scene nodeabove A's ctm_in input, if any such node exists; otherwise A is anorphan. The children of a scene node A are the scene nodes below A'sctm_out output, if any such nodes exist; otherwise A is a leaf. Theancestors of a scene node A are those scene nodes in the scene graphabove A. The descendents of A are the scene nodes in the output scenegraph of A. Scene graph nodes are described more fully in Section1.18.3.

Render traversal initiates via a Render message to a scene node andpropagates depth-first to all enabled nodes in the output scene graph(with certain exceptions described here and in Section 1.17). To renderthe full 3D scene, the application calls Render on a designated orphanroot node. Nodes may be disabled (explicitly, by the user), whichprevents these nodes and their children from being traversed duringrendering.

Parameters to the Render call specify: (1) a graphics output context, orgraphics port, to which drawing should occur, (2) the pull parameter atwhich to evaluate the expression graph, (3) the desired level-of-detailof the render, and (4) the render mode settings to be used. That is, arender call is a message:Render(gp, param, lod, Φ);

where:

gp=the output graphics port,

param=the graph parameter at which to perform the render (usually thevalue of the global animation clock, i.e., time),

lod=(lod_(geom), lod_(artic), lod_(deform), lod_(shade),lod_(light))=the desired levels of detail, and

Φ=a vector of render mode settings (described below).

When node A receives a Render, it possibly modifies the state of thedesignated graphics port in some node-specific way, issues anynode-specific geometry to that port, and then recurses over its childrenby calling Render on each enabled child. After all children havereturned from render traversal, node A reverts the graphics port to itsstate prior to A's modifications, if any, and returns to whomeverinitiated the render call on A.

The graphics port render parameter gp designates the graphics target towhich output should be directed. For interactive rendering, this couldbe an OpenGL context, for example. For rendering to an external target,such as a high-quality off-line renderer, this might refer to theinvocation context of a rendering-export plugin. “Graphics ports” aresimply a wrapper for naming such heterogenous output targets. It fallsto the output target referenced by gp to decide how to act upon thestate changes, geometry issues, etc., generated during render traversal.The gp parameter is passed unmodified through the recursive render callsto the child nodes.

The param render parameter designates the animation “time” at which thestate of the scene should be rendered. In most cases, this param will beused as the parameter for resultant Pull's and for recursive Render's ofchild nodes. Certain nodes perform input pulls or child renders atdifferent parameter values. For example, the MotionPath node (Section 4)evaluates its ctm_in input across a range of parameter values. In fact,MotionPath nodes entirely ignore the param render parameter except whenspecial path rendering options, such as tickertaping and path dilation,are enabled (Section 4.1.2).

The lod render parameter designates the levels-of-detail at which thescene should be rendered. In most cases, this lod will be usedunmodified as the lod for resultant pulls and for recursive renders ofchild nodes. A node whose bounding box projects to a small screen areamay scale lod by a value less than unity before recursing over children,providing fast, less-detailed rendering of objects that are not visuallyimportant in the rendered image. The lod may be scaled by a valuegreater than unity for the active model, that is, the model containingthe currently or most recently selected node. Automatic scaling of thelod parameter up or down is generally performed in Model nodes (Section1.18.3). In addition to automatic scaling due to visual importance oruser focus, the user may explicitly override any of the lod values atany node via an application dialog; the overridden value is used forinput pulls or child renders from that node.

The Φ render parameter designates a number of render mode settings.These settings control various aspects of interactive rendering,including: geometry style (e.g., surfaces, skeletons, bounding boxes),render style (e.g., wireframe, hidden line, flat shaded, smooth shaded),optional aids to visualizing geometry (e.g., local coordinate frames,normals, surface-curvature, texture coordinates, node names, bones), andso forth. The user may explicitly override any of the Φ settings at anode via an application dialog; the overridden Φ is used for renderingthat node and for child render calls from that node. The Φ parameter hasno affect on exports to external offline renderers.

Because the graph may contain cycles, we take care not to renderendlessly along a series of edges that lie in a cycle. Unchecked, such acyclic traversal would lead to an “infinite loop” or “infiniterecursion” and a program crash. We guard against this by raising an inRender marker flag in each node as it begins its render, and lower theflag once it has completed its render. If a node receives a rendermessage while its in Render flag is raised, it knows the render occurredrecursively via a cycle, and it will return immediately from therecursive render without traversing to its children. This is analogousto the mechanism described in Section 1.4 for ensuring cycle-safety ofgraph pulls.

The value provided at the ctm_in input of a scene node is theConcatenated Transformation Matrix (CTM) defining the coordinate spacein which that node should be rendered. This is also known as the pre-ctmof the node, since it defines the coordinate space existing before thenode makes any of its own graphics-state changes. The ctm_out outputspecifies the coordinate space in which children of a node exist, i.e.,the post-ctm of the node. That is, the post-ctm of a given node is thepre-ctm of its children. Most nodes do not modify the coordinate system,thus the value at the ctm_in of these nodes is passed through unmodifiedto ctm_out, so children of such nodes live in the same coordinate spaceas their parent. Transform nodes do modify the coordinate system, insome way transforming the matrix between ctm_in and ctm_out. The generalTransform node accomplishes this transformation by multiplying thectm_in matrix input by a xform matrix input, and providing that productat the ctm_out. Subclasses of Transform provide higher-level transformcontrols. For example, Rotate nodes rotate the ctm as specified by theirEuler-angle or quaternion rotation input. Two special subclasses ofTransform, Bones and Joints, are used to assemble articulatedhierarchies. Transforms and their subclasses, including Bones andJoints, are described in more detail in Section 1.18.3.

Several types of nodes exist to issue geometry to the graphics port. Forexample, Surface nodes are used to draw subdivision surfaces. Othergeometry nodes produce primitives such as points, lines, ellipses,quadrics and text. In addition to ctm_in and various node-specificinputs, most geometry nodes also possess a material input. This acceptsvalues of type material, which are generated by various kinds ofMaterial nodes. Basic Material nodes can be used to specify renderingstate data such as ambient, diffuse, specular and emissive colors andsurface roughness. Material node subclasses include Texture nodes forbinding a texture image map, EnvironmentMap nodes for binding areflection map, FragmentProgram and VertexProgram nodes for bindingOpenGL fragment and vertex programs, and Shader nodes for bindingprocedural shaders for external offline renderers. Material nodes aredescribed in greater detail in Section 1.18.4.

Other scene node types exist to specify the camera projection forrendering (Camera nodes) and to define scene lighting (Light nodes).Several node types exist to define named scopes over their descendentsubgraphs (Models and Parts).

There are several special kinds of scene graph nodes, and one specialsituation, that modify the above view of render traversal. Switch nodesare used to select among multiple alternate sub-scenes, each beneath adistinct ctm output. Attach nodes allow a node to exist within acoordinate space different from that of the parent's ctm_out. InverseKinematics IK_solver nodes provide an alternative joint-transformevaluation mechanism in which the transform used at a joint is derivedfrom the desired position of the end-effector of the joint/bone chain,rather than the usual case of end-effector position being determinedforward-kinematically from the joint angles. Subgraph inversion is analternative mode that inverts the usual flow of ctm data from parent tochild within a local subgraph. These special cases are discussed inSection 1.17—Dynamic Topology Changes.

1.10 Data Types

The value produced at a particular node output will be one of a numberof data types. Supported types include int, scalar (floating point, akareal), point, vector, covector (aka normal), matrix, quaternion, curve,mesh, field, deformation (Section 1.18.2), material (Section 1.18.4),image, char and string. Single- and multi-dimensional fixed-bound arraysof these types may be used as well. Anticipating the modeling languagedefined in Section 2, these may be instantiated as C-style arrays offixed bound, for example, “int[32]”, “scalar[5][6]”. A field is aparameterized type defined over a mesh; for example, “field<normal>[m]”is a surface-normal field over mesh m.

1.11 Polymorphism

While a particular node output produces data of the same type, a givennode input may accept multiple types. An output may be connected to aninput only if the type of the output matches one of the types acceptedby the input. This style of polymorphism corresponds to that of the C++programming language, in which functions may be overloaded to acceptmultiple parameter input types, but function overloading by return(output) type is not permitted.

1.12 Inputs and Outputs

Every node type defines certain input and output tabs, to which incomingand outgoing edges, respectively may be connected. An input tab canfunction in one of three modes.

-   -   1. If there is an edge incident to the input, we say the input        is connected and pulls at the input result in pulls to the        upstream node.    -   2. Alternatively, the user may specify an internal expression        for an input tab; pulls at that input result in evaluation of        the expression. The expression language is described in Section        2.2.    -   3. If an input is not connected and has no internal expression,        we say the node is defaulted, and pulls of the input yield a        default value. Nodes provide standard default values for all        their inputs, and users can specify alternative defaults.

Both input and output tabs possess a name that is unique among theinputs and outputs of that node. Input tabs specify a list of one ormore data types that may be provided to that tab (by incoming edges,expressions, or custom defaults). Output tabs specify the unique datatype output at that tab.

1.13 Gain and Bias

The user may specify a bias and/or gain for any input or output. Thevalue at that input or output will be gain*original_value+bias. Bydefault, gain=1.0 and bias=0.0 for scalar inputs and outputs. Other datatypes use appropriate values for gain and bias. For types for which theconcepts of gain and/or bias are not applicable, those adjustments areignored. Specifying gain or bias at an input scales or biases a singlegraph edge. By specifying gain or bias at an output, the user can scaleor bias all edges leaving that output. Gain and bias may be set by theuser within the application via a “node inspector” dialog box, or themodel definition code itself may specify gain and bias as described inSection 2.1.2 below.

1.14 Integrals and Derivatives

Node outputs can be integrated and differentiated with respect toparameter t. For nodes with multi-line caches, integration over[t_(a),t_(b)] amounts to summing the (clean) cache lines betweenM(t_(a)) and M(t_(b)), scaled by stepsize h=1/cr. That is:$\begin{matrix}{{\int_{t_{a}}^{t_{b}}{{f(t)}{\mathbb{d}t}}} \approx {{f( t_{a} )} + {f( {t_{a} + 1} )} + {f( {t_{a} + 2} )} + \cdots + {f( t_{b} )}}} \\{\approx {{h\quad\psi_{M{(t_{a})}}} + {h\quad\psi_{{M{(t_{a})}} + 1}} + {h\quad\psi_{{M{(t_{a})}} + 2}} + \cdots + {h\quad\psi_{M{(t_{b})}}}}} \\{{= {\frac{1}{c_{r}}{\sum\limits_{i = {M{(t_{a})}}}^{M{(t_{b})}}\psi_{i}}}},}\end{matrix}$where f(t) is the output value at parameter t, ψ_(i) is the value storedon cache line i, M(t) is the cache mapping function discussed previouslyand c_(r) is the number of cache lines per unit change in t.

Differentiating at t involves taking the finite difference at M(t).Higher-order derivatives may be computed by comparing the derivatives atsuccessive cache slots. Thus:${{{\frac{\mathbb{d}}{\mathbb{d}t}{f(t)}} \approx {\frac{1}{h}\nabla_{M{(t)}}}} = {{c_{r}{\nabla\psi_{M{(t)}}}} = {c_{r}( {\psi_{M{(t)}} - \psi_{{M{(t)}} - 1}} )}}},{and}$${{{\frac{\mathbb{d}^{k}}{\mathbb{d}t^{k}}{f(t)}} \approx {\frac{1}{h^{k}}\nabla_{M{(t)}}^{k}}} = {{c_{r}^{k}{\nabla^{k}\psi_{M{(t)}}}} = {c_{r}^{k}{\sum\limits_{i = 0}^{k}{( {- 1} )^{i}\begin{pmatrix}k \\i\end{pmatrix}\psi_{{M{(t)}} - i}}}}}},$where ∇ is the finite backward difference operator, and (_(i) ^(k)) is abinomial coefficient.

If some of the referenced cache lines are not already clean, one or morePull(t_(i))'s may be implicitly necessary. In some cases it may beconvenient to maintain a multi-line cache of the derivatives themselvesto facilitate fast evaluation of higher-order derivatives.

Expressions written in our expression language may reference theintegral or derivative of any node output as described in Section 2.2.

1.15 Signal Busses

Often many edges will follow a similar routing between two nodes orgroups of nodes. Merging these edges into a bus, drawn as a single thickedge, simplifies display of the graph. Creating a bus has no effect onthe functionality of the graph: the bussed edges continue to functionautonomously.

1.16 Transmitters and Receivers

Another common contributor to clutter in the displayed graph is theoutput that is connected to a large number of inputs. For example, theglobal animation clock will usually have an outgoing edge to theparameter input of every avar (see Sections 1.18.1 and 2.3). Transmitternodes “broadcast” their input value on a named channel. Any number ofreceiver nodes may “tune” to that channel by name, after which they willproduce the transmitted value at their output. Functionally this isequivalent to an edge that connects transmitter to receiver. Expressionsmay reference a broadcast by name (see Section 2.2).

1.17 Dynamic Topology Changes

Sections 1.3 and 1.9 described the usual way information and rendertraversal are propagated through our expression graph. As mentionedpreviously, several situations modify the usual chain of dependence andtraversal. Three such situations result from special nodes that enabledynamic local changes to the effective graph topology. A fourthsituation arises when a model dynamically changes its own root to be ata different node. We discuss each of these four situations below.

1.17.1 Switches

Switch nodes are used to select among multiple alternate sub-scenes,each beneath a distinct CTM output. These nodes may have any number ofCTM outs, and a scene node connected to any of these outputs isconsidered a “child”, but render traversal will only proceed to one suchoutput. The index of the active output is specified by a value input.The state of the switch only affects render traversal, not graphevaluation. That is, a push-dirty will push through to all connectedoutputs, and a pull from any output pulls on the inputs. The value inputaccepts both integer (or scalar) values and one-dimensional arrays ofintegers (or scalars). (Scalars will be rounded to the nearest integer.)When a single value is provided, only one output can be active at atime. Providing an array of values activates the multiple correspondingoutputs. The number of output tabs a switch node will provide isspecified by the user at model-construction time and may be anywherefrom one to an implementation-dependent fixed upper limit. A selectorvalue that does not correspond to the index of any connected outputeffectively disables all outputs.

Switches may be used to select between alternative representations or todynamically enable (i.e., show) and disable (hide) a subgraph. Anexample of the use of switch nodes to model if-then-else and multi-way“switch <selector> <cases>” constructs is given in Section2.1.9—Conditional Execution.

1.17.2 Attaches

Attach nodes allow a node to exist within a coordinate space differentfrom that of the parent's ctm_out. This is useful for a model whoseframe-of-reference changes mid-shot. For example, a model of a cupsitting on a table might logically be represented within the coordinatespace of the table; however, if a character picks up the cup, we wouldnow like to represent the cup within the coordinate space of thecharacter's hand, so that the cup will follow movement of the hand.

Attach nodes accomplish this by taking multiple alternate CTM inputs anda pair of selector inputs, and providing a single ctm_out. One selectorinput, sel_trans, specifies which CTM input will be used fortranslation; the other selector input, sel_rot, specifies which CTMinput will be used for rotation. Translation and rotation of the inputCTM's are separated through an orthonormalization step, then theselected translation and rotation are combined by matrix multiplicationand provided at the ctm_out output. Scale, shear and perspectivecomponents of the input CTM's are discarded.

Specifically, an attach node accepts a variable number of inputs:

ctm_in, sel_trans, sel_rot, target_, target_2, . . . , target_n

and provides a single ctm_out.

While the attach changes the coordinate space in which its childrenexist, attaches do not modify the scene graph structure itself: theparent of the attach is the node above the one unique ctm_in input,regardless of the setting of the selector knobs. Sel_trans and sel_rotaccept an integer or scalar input value, rounding scalars to the nearestinteger. A value in [1 . . . n] selects the corresponding target; valuesoutside [1 . . . n] select the ctm_in matrix. The user may set n atmodel-construction time to any number between zero and animplementation-dependent fixed upper limit.

1.17.3 IK Solvers

Forward Kinematics (FK) describes the normal chain of dependence withinan articulated structure in which coordinate-space transformations arespecified at a series of joints and/or bones and the position of the endof the chain (the end effector) is determined by concatenating thesetransformations. Inverse Kinematics (IK) reverses this process, allowingthe user to specify a goal position from which are computed joint/bonetransforms to position the end effector at (or as close as possible to)the goal. This computation may be over- or under_constrained, and manyalternative methods exist to solve the IK problem.

We provide a general class of IK_solver nodes that solve inversekinematic problems. Different IK_solvers implement different solutions,but all have in common a series of transform inputs and correspondingoutputs, an enable input that enables selection between FK and IKoperation mode, and inputs for the CTM's of the goal and the effector.Specific solvers may take other inputs as well. When the integer (orscalar, rounded to nearest integer) enable input is zero or negative,the IK solver is disabled, and the transform nodes it drives behaveforward-kinematically. When the enable input is greater than zero (or0.5, for scalar inputs), these transform nodes become inverse-kinematic.

Different IK solvers are capable of handling transform chains ofdifferent lengths. An example IK solver is shown in FIG. 2. This nodecan be set up to control a three-joint chain. The transform inputs in_1,in_2, in_3 are driven by the forward-kinematic control graphs for thesethree joints. The corresponding outputs out 1, out_2, out_3 areconnected to the xform inputs of the three joints. The enable input isconnected to an FK/IK switching avar. The goal input is connected to thectm_out of the target node. The effector input is connected to thectm_out of the chain end-effector node. The swing input is connected toan avar for swinging the chain through alternative IK solutions in theunderconstrained solution space. When enable is set to zero (orless-than 0.5 for a scalar input), in_1, in_2. and in_3 are passedthrough unmodified to out_1, out_2, and out_3, respectively. When enableis set greater than 0.5, internally computed transforms are provided atthe out_1, out_2, and out_3 outputs. Depending on the algorithm used inthis solver, the IK solution may completely ignore the transform inputs,or it may use the transform inputs as soft constraints and attempt tominimize error between the inputs and computed outputs. The solveralgorithms we employ are standard, widely known methods.

Note that, unlike switches and attaches, both of which are scene graphnodes, IK solvers do not take a CTM input and do not provide a CTMoutput and thus are not scene graph nodes but rather are part of thenon-scene expression graph.

The IK solvers we provide use one of several techniques. Cycliccoordinate descent (CCD) takes advantage of our ability to quicklyre-evaluate a small number of outputs following a localized invalidation(push dirty) in the upstream graph. We visit each degree-of-freedom(i.e., avar) in the chain one at a time, making an adjustment to thatavar to bring the end effector towards the goal; which way to adjusteach avar may be found experimentally by making tentative changes to theavar and Pull-ing on the end effector. Iterating on this process bringsthe effector to the goal if it is reachable, and stretches the effectorout towards the goal if it is not reachable.

Jacobian inversion solvers compute the Jacobian matrix for the linkageat the current position. The Jacobian is the multidimensional extensionto the differentiation of a single variable. It gives the partialderivatives indicating how the end effector will respond to smallchanges in the control avars. Inverting the Jacobian tells us how weshould change avars to move the end effector closer to the goal. Becausethe Jacobian only is valid locally, we make only small changes, thenrecompute the Jacobian in the new configuration and iterate. TheJacobian for a given node may be found by differentiating node outputsas described in Section 1.14. However, a more convenient way to find thecomposite Jacobian across the entire linkage is to make smalladjustments to each avar degree-of-freedom and observe the resultingchanges (deltas) in the end effector. These deltas, suitably scaled,provide the content of the Jacobian matrix for the current linkageconfiguration. In general, we will not be able to invert the Jacobian,so we use a pseudo-inverse. The Jacobian transpose method avoids the(pseudo-) inversion step by using a simple matrix transpose.

Users are free to implement their own IK solvers using the External nodeplug-in interface described in Section 1.18.6.

1.17.4 Hierarchy Inversions

Hierarchy inversion is an alternative mode that inverts the usual flowof CTM data from parent to child within a local subgraph. This occurswhen the user specifies that a different node within a given subgraphshould be used as the root of that subgraph. This has the effect offlipping edges along the path from old root to new root.

FIGS. 3 a and 3 b are examples of hierarchy inversion via subtreere-rooting. For clarity, only ctm_in and ctm_out connections are shown.In FIG. 3 a, the original subtree is rooted at A. In FIG. 3 b, thesubtree below A is re-rooted at J, and the edges along path A-C-F-J havebeen flipped and the incoming edge incident to A has been routed to J.The graph above A is unaffected.

We introduce a new message, InvertHierarchy, that nodes may pass andreceive. The application program maintains a serial number uniquelyidentifying each distinct render traversal. Before each time theapplication initiates a render, it increments the value of this serialnumber. When a node that is not inverted receives the InvertHierarchymessage, it makes an internal notation that it is part of an invertedchain during the current render pass by raising an inverted flag andrecording the serial number of the pass, and it then passesInvertHierarchy to its parent. A node that is inverted in the currentpass (i.e., has inverted raised) will ignore any InvertHierarchymessages. Once a node has been inverted, subsequent Render's within thesame pass are treated as inverted renders. When an inverted node X isrendered, it propagates the render not only to all its children but alsoto its parent. The anti-cycling in Render mechanism described in Section1.9 prevents the Render from recursively being sent back to X from itschildren. When a Render is received for a different pass, the nodelowers its inverted flag and renders normally.

To invert a local hierarchy, the user specifies the node at the top ofthat hierarchy (node A in FIGS. 3 a and 3 b), the node to become the newacting root (J in the Figs.), and the animation time range over whichthe hierarchy should remain inverted. Each node maintains a list ofinversions that initiate at that node. In FIGS. 3 a and 3 b, node A willrecord internally that within the interval [t_(invert), t_(restore)] thesubtree (properly, subgraph, since it may contain cycles) below it isre-rooted at node J. When A receives a Render at a parameter value inthis interval, it will first raise its in Render flag (Section 1.9) andthen will send InvertHierarchy to J and by induction to F and C. A willthen pass the Render to J, which will recursively render F and C (and K,E, H, and I). A will receive a Render from C but will ignore it becauseA's in Render is raised; however, A will note that it received theRender from C. When J returns from rendering, A will proceed withrendering itself and then will recurse over all its children except C,that is, over B and D.

When rendering at a node on the inverted path from J to A, the behaviorof the ctm_in and ctm_out inputs and outputs are switched; that is, itis as though ctm_in had become ctm_out, and vice versa. Rendering ofgeometry will still take place in the pre-ctm of the node; however inthis case the pre-ctm first needs to be computed from the post-ctm. Forinverted non-Transform nodes, this involves just copying the ctm_out tothe ctm_in. For inverted Transform nodes, the post-ctm from the ctm_out“input” is multiplied by the matrix inverse of the xform input and theresulting pre-ctm is provided at the ctm_in “output”. This reverses theusual (non-inverted) behavior of Transform nodes, which multiply ctm_inby xform, yielding ctm-out. The behavior of nodes not on the invertedpath is unchanged.

Hierarchy inversion may be used when an animation task calls for aninverted view of a model hierarchy. For example, a character hanging bythe arm from a tree limb may be animated relative to a model root in herhand rather than the usual model root at her pelvis. Bending the elbowjoint of that character will then cause her entire body to swingrelative to her stationary forearm, rather than the other way around.

1.18 Expression Nodes

Data in the graph originates from several types of nodes, includingparameter drivers, constants, file readers and avars. Many other nodetypes exist that process this data, performing calculations, generatingother data, and producing the geometry, lighting and shading thatultimately is rendered in the 3-D view window. Here we describe eachmajor category of node and give examples of each.

1.18.1 Data Sources

Parameter drivers, such as clocks, have already been mentioned. Suchnodes have no inputs (or optionally min/max range inputs) and only asingle output. They may be configured internally to vary their outputover some value range. For clocks, the rate at which they vary is oftentied internally to the application's notion of “wall-clock” time.However clocks may be paused, run backwards, run at slower- orfaster-than-real-time speeds, or single-stepped (that is, incremented bysome fixed stepsize) forwards or backwards. Non-clock parameter driversalso vary over some parameter range, but conceptually they take on allvalues in that range simultaneously. In practice, they return zero inresponse to a pull for kInputParam, and otherwise take on whateverspecific value was pulled, clamped within their allowed range, if any.

Constants are another node type. As their name implies, constants havean unchanging value, no matter at what parameter value they are pulled.The value is set internally to the node at model-construction time. Theyhave a single value output, and no inputs. There is no conceptualdifference between providing a constant-valued input expression orconnecting an input to a constant node. In different situations, one orthe other method may be more convenient.

File readers are similar to constants, except that rather than theirvalue being set internally, it is read from a disk file. File readerstake a file input string designating the file path and provide a valueoutput. Various file readers are available for reading different typesof files, and some may have additional inputs for configuring importoptions or additional outputs providing extra information about the dataread. For example, MeshReader nodes read meshes saved in Wavefront OBJand several other formats. This reader has an objects input accepting astring giving names of objects to be read in that file; if no value (ora NULL value) is provided, all objects in the file will be imported.Other inputs control the assembly of meshes from the OBJ data, forexample, by specifying a threshold dihedral angle at which a hard edge(two distinct face normals) should be introduced in the mesh normals.The MeshReader provides a mesh or array of meshes at its value output,plus an objects_read output giving the names of the meshes read. Otherfile readers exist for loading numerical arrays, images, QuickTimemovies, audio, motion-capture data, and so forth.

Avars are nodes that correspond to articulated variables as defined inW. T. Reeves, E. F. Ostby, and S. J. Leffler, The menv modeling andanimation environment, Journal of Visualization and Computer Animation,1(1):33-40, August 1990. Examples of articulated variables arequantities that a human operator might want to vary over time (or otherparameter value for a non-clock parameter driver). They have a parametervalue input, which usually comes directly from the global animationclock node; an “override” input to allow overriding the data set in theavar; and “value” and “solo” outputs. The value output gives the valuethe avar takes on at the input parameter (when the pull is atkInputParam) or at the parameter requested by a pull. The solo output isdiscussed in Section 5.

As described in Section 2.1.3, avars are a templated node type and canbe instantiated for any value type for which certain basic mathematicaloperations are defined. We have found scalar- and quaternion-typed avarsto be of particular value, but other types are possible as well.

The value output of an avar is a function of the param input. Thefunction definition is complex and is described in detail in Section 5.The function definition may include kinematic, dynamic, procedural,sampled and stochastic components. The kinematic components of an avar,if any, are defined by parameterized one-dimensional piecwise-polynomialcurves, or splines, with knots specifying the value at particularparameters along the spline. The form and specification of these splinesare described in Section 2.3.

The application keeps track of a user-specified current manipulationmode. Manipulation modes correspond to common modalities ofmanipulation; for example, translation, rotation, scale, bend,squash/stretch, mesh deformation, global deformation, etc. The activeavars at any given moment are those avars designated (by the user atmodel-construction time) as relevant to the current manipulation mode,and that are within a subgraph driving any input (other than ctm_(in))to the currently selected scene graph node or nodes. Active knots areknots in the active layer (see Section 5) of the active avars that areunlocked and free to be manipulated (Section 5.3).

We provide an in-camera indication of the names of the currently activenode and the active avars. We also indicate in-camera which active avarshave active knots at the selected motion-path knots (see Section 4.1.3)or current frame if no path knots are selected.

In the top-left of FIG. 10 a, text annotations indicate that the activeavar is the “rot” avar of the “root_joint” node of the “Rod” model. Theyellow box around the word “rot” indicates that this avar has a knot atthe current frame (frame zero). The yellow diamond on the motion path atframe zero confirms the existence of a knot there (Section 4.1.3), asdoes the yellow solid-line box around the large frame number “0” in thelower right corner of the view. If there were additional avars active,they would be listed to the right of the word “rot”, and each would havea yellow box around it if and only if that avar had a knot at framezero. The box around the large frame number at the bottom right would bedrawn with a solid line if all active avars had knots at frame zero; ifonly some active avars had knots at frame zero it would be drawn with adashed line.

In FIG. 10 b, we see that the “rot” avar is still active, but that itdoes not have a knot at the current frame (frame nine) because the word“rot” is not boxed in yellow. Similarly, we note there is no yellowdiamond on the motion path at frame nine, and the large frame number “9”at the bottom-right of the view has no box around it.

Each avar is internally configured at model-construction time to havesome default value, which is the value that avar takes on when it has noanimation (i.e., no motion channels) in any enabled layer (see Section5). Several settings concerning the preferred display format for avardata may be specified, such as the unit of measurement (e.g., meters,kilograms, seconds, radians, degrees), linearity (linear vs.logarithmic), and scale. An avar may also be configured to have anallowed output value range (i.e., hard limits) and a preferred range(soft limits). These ranges do not affect the evaluation of the avaritself, but they can be queried, for example, to guide a constrainedoptimization solver running in a dependent node or an externalcomputation. The user can optionally set direct manipulation controls(see Manipulators below) to respect these value limits. For scalaravars, soft- and hard-limits are specified as minimum and maximumvalues. For quaternion avars, soft- and hard-limits are each specifiedas an array of unit quaternions demarcating a convex “spherical polygon”within which the avar may move.

The user can enable a Hold Poses mode in which all time-varying avarsare evaluated as though they contained in every layer only stepped knotsat those times at which the active avars have knots in the active layer.This has the effect of arresting movement between active knots. Whenplaying the animation in this mode, at each active knot the sceneelements jump discontinuously to the positions and state they normallytake at that time, and otherwise do not move. We accomplish this bysubstituting for the output value of the global animation clock the knottime t_(held) of the last active knot prior to or at the true animationtime. Since time-varying avars use this output value as their paraminput, in response to pulls at kInputParam during rendering, they willhold their own value outputs constant except when the animation timet_(now) passes the next active knot time t_(next) _(—) _(hold); thent_(held) will update to t_(next) _(—) _(hold), and all the avars willjump to the output state at the new t_(held).

1.18.2 Operators

Operators produce an output that is functionally dependent on somenumber of inputs. For example, basic operators include thepolymorphically typed nodes Multiply and Add. These are actuallyconvenience wrappers around a general Compute node that takes anarbitrary number of inputs and a user-provided string that expressessome function. We evaluate the function on the input tabs and providethe result at the output tab. The expression language is described inSection 2.1.5. Other convenience wrappers include Translation, Rotationand Scaling, which take scalar inputs for tx/ty/tz, rx/ry/rz andsx/sy/sz, respectively, and produce a transformation matrix output.

IK Solvers are expression nodes for introducing dynamic changes in graphtopology in support of inverse kinematics. They are described in detailin Section 1.17.3.

Deformers are nodes that compute some point-valued function of points.That is, a deformer expresses the function p′=f(p). Mesh deformers applythis function across the vertices of a mesh, usually in some localcoordinate frame, for example, a body coordinate frame, or in asurface-relative fashion. Spatial deformers, or global deformers, applytheir function at arbitrary positions in global or local space. Bothtypes of deformers accept a mesh_in input mesh or point array. The onlyreal difference between mesh and spatial deformers is that, while eachaccept additional parameters controlling the deformation function, theinput parameters of mesh deformers are point-wise fields that conform tothe input mesh, while spatial deformers and their inputs have noknowledge of, and need not conform to, the point-data topology orlayout. (The one exception to this is that spatial deformers, like meshdeformers, can subdivide mesh inputs before acting on them, as describedbelow.)

All deformers provide two outputs: mesh_out and deformation. Deformersmay be used in either of two ways. They may be used to actually deformtheir input mesh or point array in a vertex-wise or point-wise fashion,with this deformed mesh or point array available at mesh_out.Alternatively, they may be used to compute a vector field of the(global, local, or surface-relative, depending on the type andconfiguration of the deformer) vertex- or point-wise displacements ascomputed by the deformation function; this vector field is available atthe deformation output.

In our implementation, all meshes are represented and rendered assubdivision surfaces. When a mesh is provided to the mesh_in input on adeformer node, the refinement level at which the deformer is acting maybe specified at the deformer's sdlevel input. The surface will bedeformed by moving the refined vertices at the specified level ratherthan moving vertices at the base (unsubdivided) level zero. Whenspecifying sdlevel >0 for a mesh deformer, point-wise parameter fieldsconform to the subdivided mesh. Sdlevel is ignored when deforming simplepoint arrays lacking connectivity information.

Many deformer subclasses exist, such as RotateDeformer, BendDeformer,CylindricalDeformer, SphericalDeformer, ConicalDeformer, MuscleDeformer,JointDeformer, BoneDeformer and numerous others. Each performs aparticular type of mesh deformation. These deformers may be chainedsequentially, passing the mesh_out of one deformer to the mesh_in of thenext; or their deformation vector-field outputs may be combined (summed)in parallel and later applied to a mesh via point-vector addition. It iscommon for complex deformation networks to include a mix of parallel andsequential deformations.

The SkeletalDeformer mesh deformer takes as inputs any number (up tosome implementation-dependent limit) of bone ctm_out's, plus four pairsof bone_id, bone_weight point-wise scalar-field inputs and a body_spaceCTM input. The bone ID's index (counting from one) into the bone inputs,associating up to four bones with each point in the input mesh (or pointarray). The body-space CTM (pre- or post-, as described below) of eachbone is noted with the skeleton in its home (default) pose, then thetransform that carries each bone from home pose to the current pose isscaled by the corresponding bone weight and used to transform each meshpoint. For each point, up to four such weighted-bone transforms areapplied sequentially. A positive bone ID selects the post-ctm of theindexed bone, while a negative ID selects the pre-ctm of the boneindexed by the absolute value of the ID. Specifying a bone ID of zero isequivalent to specifying a zero weight. This deformer thus performsfour-way weighted-bone deformation of the input mesh as is popular inmany consumer-level and real-time graphics applications. On hardwarethat supports the OpenGL capabilities GL_ARB_vertex_blend orGL_vertex_program, we are able to perform the skeletal deformation onthe graphics card, provided (a) that the number of bones falls withinhardware limits and (b) that the deformed mesh is not needed for anysubsequent computations (other than rendering). For convenience, whenconnecting a SkeletalDeformer, one need only specify the root node ofthe skeleton hierarchy and the application will take care of connectingthe body_space input and the many bones in the skeleton (and will markthose bone input tabs hidden so that they and their incident edges arenot drawn in the graph editor view, thereby reducing clutter).

1.18.3 Scene Graph Nodes

Scene graph nodes were described in general in Section 1.9. These nodesconstitute the scene graph embedded within the larger expression graph.While there are many varieties of scene graph node in our system, allrespond to a Render message, and all take a Concatenated TransformMatrix (CTM) ctm_in input and provide a ctm_out output. The Rendermessage and the roles of ctm_in and ctm_out in render traversal aredescribed in detail in Section 1.9. There are numerous subclasses of thebasic SceneGraphNode class, the most important of which are describedbelow.

Group nodes are the most basic type of scene graph node. A group nodeitself serves only as a collective parent for the children beneath it.

Part nodes introduce a named scope over their scoped subgraph. We definethe scoped scene nodes of a part node P as those scene-node descendentsof P reachable by a path, such path having length greater than zero andcontaining no parts except one or two at the endpoints of the path. Wesay the scoped scene nodes of P are “scoped to P”. The non-scenesupergraph of a scene node R we define here as the union of non-scenesubgraphs above (i.e., ancestral to) R's inputs (other than ctm_in). Thepotential scoped subgraph of part P we define as the union of the scenenodes scoped to P plus the nodes in the non-scene supergraph of anyscene node R scoped to P that are reachable from R via a path containingno scene nodes except R. Then we define the scoped subgraph of part P asthe union of (a) those nodes in P's potential scoped subgraph that donot belong to any other part's potential scoped subgraph plus (b) thosenodes below P, or in the non-scene supergraphs above P's descendents,that are not in the scoped subgraph of any descendent of P (other than Pitself) and that are not in the potential scoped subgraph of any nodenot descendent to P.

The names of part nodes in the scene-graph path from the root to a givennode A contribute the path components to A's full path-name. Forexample, a node “Thumb” whose full path is “/Fred/RArm/Hand/Thumb”exists in the scoped subgraph of a part node “Hand”, and “Hand” is inthe scoped subgraph of part “RArm”, which in turn is in the scopedsubgraph of part “Fred”, which itself is in global scope.

A model is a type of part node that designates the top scope of anindependently loadable scene subgraph. In the above example, part “Fred”would be best represented as a Model node.

Most scene graph nodes pass their pre-ctm (from ct_min) through to theirpost-ctm (at ctm_out) unmodified. Transforms are scene graph nodes thattransform their pre-ctm to produce their post-ctm (Section 1.9). BasicTransforms multiply their ctm_in input by the transform matrix at theirxform input and assign the result to ctm_out. Convenience subclasses oftransform, such as Translate, Rotate and Scale, simplify building commonexpressions, here effectively combining a Translation, Rotation orScaling node, respectively, with a Transform node. The Rotate node ispolymorphic, accepting any one of rx,ry,rz Euler angles, an orthonormalmatrix, or a quaternion. Two Transform subclasses, Bones and Joints,provide higher-level control over the matrix transform and enable directmanipulation via Manipulator nodes (discussed below).

Bones are transform nodes that represent a nominally rigid linearelement within an articulated skeleton. Inputs allow for animation ofprismatic (lengthwise) translation, revolute axial twist, and twoorthogonal axes of bend. There are independent min_u and max_U inputsfor controlling the section of bone over which bend and twist occur. Itmay be convenient to normalize the length and other animation parametersof bones so that, for example, setting all length avars to 1.0 willproduce bones of length appropriate for the default pose of a character.This may be accomplished by adjusting the bias (and possibly gain) ofeither the bone inputs or the avar outputs; generally it is best to setthe bias and/or gain at the avar output so as not to interfere withother processes (e.g., IK solvers) that may try to adjust the boneinputs directly (see Section 1.17.3). Similarly, manipulation limits maybe set by specifying value limits on the avars.

Joints are transforms that represent revolute or spherical joints withinan articulated skeleton. Joints accept as their xform input either anorthonormal matrix (e.g., from an Euler-angle rotation) or a quaternion(e.g., from a quaternion-valued avar). It is often desirable to assemblethe model so that all joints will be centered (i.e., Euler-rotationavars zeroed, quaternion avars set to quaternion identity) when theskeleton is in the default pose. As with bone parameters, this can bedone by setting a “pre-orientation” via bias and/or gain at the avaroutputs (or, less ideally, at the joint rotation input). Joint limitsare set via limits on the (scalar or quaternion) avar(s) driving thejoint. The application provides a simple “by-demonstration” joint-limitconfiguration mode in which the user moves the joint through its allowedrange of motion and the application sets corresponding avar limits. Forscalar avars, this is straightforward; for quaternion avars, we fit aconvex “spherical polygon” around the observed motion range. The usermay subsequently edit these joint limits “in-camera” by interactivelymanipulating the unit-quaternion vertices of this spherical polygondrawn on a 3-D virtual sphere centered on the joint, or the user mayspecify new limits by demonstrating a new range of motion.

To facilitate user understanding of the bones and joints comprising askeleton, we provide an application option to draw a translucentskeleton overlay over geometry (FIGS. 10 a and 10 b). This isimplemented by rendering a ghost (Section 3) in skeleton mode at thecurrent animation-clock time. Bones and joints normally are excludedfrom final (off-line) rendering.

Surfaces provide a mechanism for rendering meshes as Catmull-Clarksubdivision surfaces. A surface node accepts a mesh input and a materialinput (described below). The mesh undergoes n Catmull-Clark refinementsteps, where n equals the numerical floor of the effective lod (Section1.7) of the render. Commonly, a mesh will originate at a MeshReadernode, be passed through a deformer network, and ultimately be fed to asurface node for rendering.

Switches and Attaches are scene graph nodes for introducing dynamicchanges in scene topology. They are described in detail in Section 1.17.

Glyphs are scene graph nodes providing notational graphics that may bedisplayed, and possibly manipulated, in interactive views but that arenot really part of the CG scene and are excluded from final (off-line)rendering. Examples include MotionPath (motion-path curves), ControlHull(meshed-surface control hulls), EditPoint (control points on motionpaths and control hulls), Annotation (text labels and 2-D graphicalmarkup), Jack (a 3- or 6-d of point for use as an IK goal, attachtarget, or anywhere else a virtual reference point is desired), andManipulator (discussed below). During a normal rendering pass, glyphsare rendered in the same manner as other scene graph nodes, but glyphsgenerally are excluded from ghost and shadow rendering passes (Section3).

Manipulators are glyphs that provide in-camera direct-manipulationcontrol over animatable elements of the 3-D scene. Example manipulatorsinclude an arcball rotation controller, x/y/z translation handles, x/y/zscale handles, and a bend manipulator handle for controlling benddeformers. The application is responsible for creating and deletingmanipulators appropriate to the surface, bone, joint, or other objectcurrently selected and the current user-specified “manipulation mode”(defined above). For example, if the user selects a joint and thenchooses “rotation mode”, an arcball controller will be created by theapplication and connected as a child of the joint node. When the userlater deselects the joint or changes to a different manipulation mode,the application will delete the arcball controller.

Light scene graph nodes define theatrical lighting in the 3-D scene.They take inputs corresponding to the particular light type. Forexample, Pointlight has inputs for diffuse and specular color, falloffrate, and linearity, while Spotlight also has inputs for cone angle,shape and penumbra. Like surfaces, lights take an optional materialinput for creating advanced effects such as gobo's (projected textures)and custom beam distributions or for associating arbitrary shaders witha light (Section 1.18.4). An n×m lighting matrix defines the weight withwhich each of the n lights in a 3-D scene illuminate each of the msurfaces in the scene. This matrix defaults to 1.0 everywhere and iseditable by the user in an application dialog. Each light's intensity isscaled by the corresponding matrix entry before final rendering (andoptionally before interactive rendering) of each surface.

Camera scene graph nodes define the virtual cinematic cameras used forviewing a scene. A camera defines the viewer's eye-point. The basiccamera class takes transform-matrix inputs projection and view and setsthe rendering projection and view matrices accordingly. Subclasses ofthe basic camera exist for providing higher-level control over cameramount movements (e.g., dolly, truck, crane, pan, tilt, and roll),view-camera movements (rise (and fall), shift, swing and tilt), lenssettings (focal_length, aperture, distortion), shutter (shutter_angle)and film transport (fps). A filter input accepts a material value (seebelow) for creating advanced effects such as lens filters, vignetting,chromatic aberration, internal reflections (flare) and subsurfacescattering (bloom), as well as binding arbitrary shaders to the camera.Cameras provide an image_plane output optionally used in conjunctionwith Film Backs and Imagers (Section 1.18.5) to further define orutilize the rendered image produced.

1.18.4 Material Nodes

Material nodes define optical and material properties such as ambient,diffuse, specular and emissive color; surface roughness (“shinyness”);texture, glow, reflection, irradiance, bump and displacement maps;atmospheric effects (“participating media”), and so forth. The simpleColor material subclass provides a convenient way of setting justdiffuse color. Texture material nodes bind a 1-D, 2-D, or 3-D texturemap to a (OpenGL or offline renderer) texturing operation or the mapinput of another material node. EnvironmentMap nodes bind a spherical orcube reflection map. The texture image data itself comes from an imageinput, which may originate at a FileReader (Section 1.18.1) or at anImager (Section 1.18.5).

Programmable materials include the FragmentProgram and VertexProgramnodes, which bind OpenGL fragment and vertex programs. Shader nodesallow a named external “shader” to be bound for use during final(off-line) rendering by a batch rendering program such as Pixar'sRenderMan.

All materials take a material_in input and provide a material_outoutput. Connecting multiple materials in series allows the assembly ofcomplex combined effects. For example, several texture nodes might beconnected, with one specifying a diffuse map, another modulating thatdiffuse map with another map, a third specifying a glow map, and a forthspecifying a bump map. These texture nodes could be connected to ashader for controlling the off-line rendering process and a basicmaterial for specifying underlying material properties such as diffuseand specular color. The effects of these nodes will be combined as eachis chained together.

An Atmosphere node is provided as a proxy for accepting a material inputto affect the scene atmosphere, providing participating media effectssuch as haze, fog, Rayleigh scattering (visual perspective) anddiffraction effects (e.g., halos and coronas). A given scene may have atmost one atmosphere node that has global effect over the scene.Atmosphere material is primarily of importance for off-line rendering;with the exception of fog (which is supported in hardware by currentgraphics cards), most atmosphere materials are ignored duringinteractive rendering. Atmosphere nodes are unusual in that they have nooutputs.

1.18.5 Imaging Nodes: “Camera Backs”

Film backs may be used in conjunction with cameras to provide furthercontrol over the final-image generation process. Film backs accept animage lane input from a camera and a material input that can be used tosimulate properties of film stock such as speed, transfer function,grain and reciprocity failure. Film backs also accept inputs forconfiguring the film format (aspect ratio and anisotropy). Film backsare the only node type other than atmosphere nodes that produce nooutput value. With the exception of the aspect-ratio setting, which theuser may apply in constraining camera-window resizing during theinteractive session, film backs are used only for controlling externaloff-line renderers and have no effect on interactive rendering.

Imagers accept an image_lane input from a camera and produce an imageoutput. Imager nodes are analogous to the digital-imaging backsavailable for traditional (physical, film) cinematic cameras. They aretypically used in conjunction with texture nodes (Section 1.18.4) toproduce a reflection map or otherwise capture a rendered image of thescene for use within that scene. The image output may also be directedto an export node (implemented as an External, see Section 1.18.6) forintegration into editing, compositing or color-correction software.

1.18.6 Miscellaneous

Transmitters and receivers together provide graph-wide broadcasts asdescribed in Section 1.16.

Macros are encapsulated graphs in which particular outputs andunconnected inputs of the graph have been designated “public”. All otheroutputs and inputs are private. The user may instantiate the macro inanother graph, where it will appear as node providing these publicinputs and outputs. Macros do not alter evaluation of the total graph atall; they only provide the user convenience of an encapsulated view ofgraph structure.

Externals are nodes whose definition has been loaded through an externalplug-in interface. They are coded in C++ by using a provided API, thenare loaded at runtime and are available for use along with macros andbuilt-in nodes.

1.19 Graph Editing

FIG. 4 is a screen shot of an exemplary control graph 402 for the bottlemodel shown in FIGS. 8 a-8 c. The graph editor 400 can be used forcreating, viewing and editing the control graph 402. Example nodes 404are shown in FIGS. 2, 4 and 6. Edges 406 are displayed as polylinesjoining node output tabs 408 and input tabs 410. Nodes 404 may beinstantiated by selecting from a list of known node types. User-creatednodes (Macros and Externals) also appear in this list. After creating anode 404, the user may click-and-drag on the node 404 to reposition itwithin the control graph 402. The user may connect node outputs to nodeinputs by dragging a rubber-band line from an output tab 408 to atype-compatible input tab 410, or vice versa. In some implementations,selecting one or more nodes 404 or edges 406 and pressing the Delete keywill delete the selected graph elements. Selected nodes 404 may also becopied and pasted to create new nodes 404 of the same type with the sameinternal settings. Internal node settings (e.g., input expressions, gainand bias, user comments, output caching strategies, etc.) may be viewedand edited either in pull-out drawers 600, as shown in FIG. 6, and/or anode inspector application dialog. In some implementations, selecting anode 404 in the control graph 402 selects that node 404 elsewherethroughout the application user interface, and vice versa.

Nodes 404, input and output tabs 410, 408, and edges 406 may be hidden,which has no effect on their behavior but prevents them from being drawnin the graph editor. Hiding an input tab 410 or an output tab 408effectively also hides edges 406 incident to that tab. Hiding a node 404hides its connection tabs and incident edges as well. Certain connectiontabs are hidden by default. For example, Cameras, Lights, and basicMaterial nodes accept many inputs, but commonly only a few are used, soby default the more obscure inputs are hidden. Similarly,SkeletalDeformer nodes accept a great number of inputs, one from eachbone within a skeletal hierarchy, but these bone inputs are all hiddenby default. This provides a less-cluttered view of the graph. The usermay show or hide any node, tab or edge at runtime or model-constructiontime. The user may also elect to have hidden elements shown within thegraph editor 400, for example, to allow selecting a hidden element.Hiding a node in the graph editor 400 hides that node in 3-D sceneviews, and vice versa. Control graphs 402 may also be understood andedited with an external text editor using the text-based programminglanguages described next. FIG. 5 shows nodes 500 a, 500 b and 500 c,connected with polylines 502 a and 502 b.

2. Programming Languages

Expression graph structures may be built interactively via a visualgraph editor user interface. Frequently it is more convenient to specifythe graph via a text-based programming language. We describe the core ofour model-definition language (Section 2.1). Two subsets of the modelinglanguage are commonly used outside the model-definition setting: theseare the syntaxes for expressions, described in Section 2.2, andanimation, described in Section 2.3. While we describe these in threedistinct sections, they all are properly part of the same language andall can be used to define a model or collection of models. The resultinglanguage is Turing complete, allowing specification of any computationor algorithm that could be expressed in a general purpose programminglanguage such as C.

2.1 Modeling Language

The modeling language in our system is an object_oriented C/C++-likelanguage. It has POEM (Parse Once, Execute Many) execution semantics,with models being unrolled and the corresponding graph instantiated onceat parse time, after which the same graph is evaluated as often asnecessary. The system provides garbage collection: there are no “new” or“delete” operators. Named symbols in the language correspond to nodes,node inputs, and node outputs. Common preprocessor directives such as#include, #ifdef, #elif, #else and #endif are available. We also providea #includeonce convenience directive for including a particular headeronly once within a given translation unit.

2.1.1 Declarations

A node is created in the language simply by declaring it. For example,the model code: Model Joe; Model Sue;

creates two nodes of type Model named “Joe” and “Sue”. Node inputs maybe specified within parentheses on the declaration line as positionalparameters or as name=value pairs: Translate sideways(5,0,0); Translateupwards(ty=2); /* tx and tz inputs default to 0.0 */

The first line above creates a Translate node “sideways” with its inputtabs tx, ty and tz set to constant values 5, 0 and 0. The second linecreates a node with only one input set to a non-default value. Anonymousnodes may be created as well, for example, “Translate(1,2,3).” Here thesystem will generate a symbol name to be used for the node, however thatsymbol name will not be available to the program so there is no way torefer to that node subsequently.

2.1.2 Inputs and Outputs

Node inputs and outputs may be referenced via the selection operator(“.”, period), either by name as node_name.tab_name or by index asnode_name.in[i] (for inputs) or node_name.out[i] (for outputs). Thefirst output of a node may be referenced with just the node name alone:thus, “myNode” is equivalent to “myNode.out[0].” Inputs are lvalues andoutputs are rvalues. Assigning an output to an input establishes anedge. An edge is also created when an output is passed as an inputparameter to a declaration: Avar up, right; Translate trans (ty=up); /*connect up.value -> trans.ty */ trans.tx = right; /* connect right.value-> trans.tx */

Nodes and inputs/outputs inhabit distinct namespaces, so we could havewritten the above as: Avar tx, ty; Translate trans (tx=tx, ty=ty,tz=nil);

The predefined symbol “nil” (or equivalently “NULL”) indicates we wishto leave input tz disconnected. This is commonly used with positionalparameters, for example, “Translate (tx,nil,tz)”.

When referencing an output (as a parameter or as the rvalue in anassignment), by default that output will be Pull-ed at the param and lodat which the causational pull occurred. (The causational pull is thepull that caused the node on the left of the assignment operator tore-evaluate and thus request the given output rvalue). We can pull at aspecific param and/or lod via the function-call-like format output(param, lod). Given an avar ty, “ty(2.5)” (or “ty.value(2.5)”,“ty.out[0](2.5)”, “ty (param=2.5)”, etc.) references the value of ty atparam=2.5. “Surface coarse (mesh (lod=0.5));” creates a Surface node“coarse” from the output of node “mesh” at level-of-detail 0.5.

In addition to its input tabs, every node has an implicit parameternamed “comment” that can be used to provide text for the node comment.Avar nodes may have default values (the value the node takes when noanimation channels are present) specified via assignment. Avarrx(policy=”+”, /* additive policy: see Section 5 of text */units=”degrees”, /* only affects display format in GUI */comment=”rotation about x axis”) /* visible in GUI help */ = 30.0;    /* defaults to 30 degrees */2.1.3 Avars and Parameterized Types

Several node types are in fact templates, and the template typeparameter may be defaulted. For example, Avars are a template withdefault type “scalar”. (The type “real” is a synonym for “scalar”.) Toinstantiate an avar with a different type, a C++-like syntax is used: Avar tx, ty, tz; /* scalar avars */  Avar<scalar> sx, sy, sz; /* morescalar avars */ Avar<quaternion> rot; /* quaternion avar */

By default, scalar avars default to 0.0 and use an additive layeringpolicy. (Layering policies are described in Section 5). Quaternion avarsdefault to quaternion identity (x,y,z,w=0,0,0,1) and use amultiplicative layering policy. The 0.0 default and additive layeringprobably are not what we want for scaling avars (e.g., sx,sy,sz above).We can specify a different policy and/or default. The default isinferred from the policy if not specified: additive avars default to atype-appropriate zero and multiplicative avars default to identity. Avartx;   /* will default to 0.0, additive policy */ Avar length=1; /* willdefault to 1.0, additive policy */ Avar sx(policy=”*”);   /* willdefault to 1.0, multiplicative policy */ Avar<quaternion> rot; /* willdefault to identity, multiplicative policy */

Implicit in the declaration of an avar is assignment of the globalanimation clock output to the avar param input. The connection is madevia broadcast (i.e., through a transmitter/receiver pair) to reduceclutter in the graph editor. If we do not want the avar to betime-driven, we need to explicitly specify that the param input is nilor is connected to some other output, as shown in Listing 3 below.

We may specify animation for the avar within a scope block (“{ . . . □”)following the avar declaration using the format described in Section2.3. Normally animation for time-variant avars would not be specifiedwithin the model itself; however, there are situations in which it isconvenient to include “animation” within the model. For example, an avardriven by the u-parameter of a 3D curve might be used to define theprofile of an extrusion of that curve. Similarly, an avar could specifyone coordinate of the profile curve for a surface-of-revolution. In suchcases, we may well wish to include the avar data within our model code.An example showing the use of embedded avar data is given in Listing 3below.

Other templated node classes include Constant, Compute, Transmitter andReceiver. When the type can be inferred from usage we need not specifyit explicitly: Constant name=”Fred”; /* <string> is inferred */Transmitter foo (name); /* <string> is inferred */ Transmitter<string>bar; /* can't infer type so it is specified */ bar.input = name;Receiver recv (bar); /* <string> is inferred */2.1.4 Assembling Scene Hierarchies

Scene node hierarchies may be built up by enclosing children within ascope block. Transform nodes implicitly scope the nodes beneath them upto the point where the transform itself goes out of scope. Materialnodes implicitly chain into an input subgraph above the “material” inputtab of subsequent geometry until that material node goes out of scope.Thus: Translate (1,0,0) { Cube (1,1,1); } Sphere (1); and { Translate(1,0,0); Cube (1,1,1); } Sphere (1);

both translate the cube, but not the sphere, to the right. Similarly, wecan define a simple Box model with animation controls as: Model Box {Avar tx=0, ty=0, tz=0; Avar sx(policy=”*”)=1, sy(policy=”*”)=1,sz(policy=”*”)=1; Avar<quaternion> rot; Translate(tx,ty,tz);Rotate(rot); Scale(sx,sy,sz); Avar r=1, g=1, b=1, a=1;Texture(”mytex.tiff”); Color(r,g,b); Cube(1,1,1); }

The cube primitive will be transformed, textured and colored by thestatements that precede it. The close of scope at the end of the modelensures that changes to the transformation stack and material chainswithin model Box will not affect geometry outside of this model.

2.1.5 Expressions and Optimization

Compute nodes may be created explicitly like other nodes or implicitlyvia an expression. The model program may contain arbitrary expressionsas defined in Section 2.2 below, with the one modification that certain“predefined locals” are accessed as parameter-less functions (e.g.,“lod( )” and “ctm( )”). Standalone expressions assigned to basic types(int, real, string, etc.) implicitly create a Compute node (except whereoptimized away, as described below). Assignment of expressions to nodeinputs may be made with the assignment operator (“=”). For example: reala = 3 + 4 * time; /* create a Compute node named ”a” */ Scale s (a+1,sin(2*a), 1); /* create a Scale node ”s” with input expressions */ Avarup_down; Translate trans; trans.ty = 2 * up_down; /* connect edge andset input expression */

These expressions and the implied graph structure may be optimizedautomatically. For example, multiple confluent expressions may becombined into a single Compute node (corresponding to the traditionalcompiler optimizations of “procedure integration” and “copypropagation”). When possible a standalone expression whose result isused only once will be encoded into the destination node input ratherthan instantiated as a separate Compute node (“inlining”). Commonsubexpressions may be merged into a single Compute node or inputexpression (“common subexpression elimination”). Expressions whoseresults are not needed are dead-stripped (“dead code elimination”)unless their declaration is qualified by the volatile keyword: {volatile char c = ’a’+1;/* c is unused but will not be stripped */ }

The function within an explicitly declared Compute node is specifiedfollowing an assignment operator (“=”), using the expression languagedescribed in Section 2.2. Compute input names are scoped within thefunction definition. For example, we may (re-)define the linearinterpolation function for meshes as:

Compute myMeshLerp(real x=blend, Mesh a=M0, Mesh b=M1)=(1−x)*a+x*b;

As x goes from 0 to 1, myMeshLerp.out will linearly blend from M0 to M1.(We define mesh-scalar multiplication as a scaling of the vectordistance of mesh vertices from the local origin.) Specifying type nameshere is optional. We can define a more general lerp (behavingidentically to the built-in lerp function) as:Compute myLerp(x=, a=, b=)=(1−x)*a+x*b;

We specify “x=” (shorthand for “x=nil”) to indicate that “x” is thefirst input to myLerp, rather than the name of a referenced output ofsome other node. The above statement will create a Compute node withthree disconnected inputs named “x”, “a” and “b”. Input and output edgeconnections to this node are legal to the extent that the data typesalong those edges yield a legal expression. For example, passing(scalar,scalar,scalar) and (scalar,mesh,mesh) are both valid because theresulting expressions are valid. However, passing (mesh,scalar,scalar)will produce an error because mesh-scalar subtraction (the “1−x” above)is undefined.

2.1.6 Macros

We may define a macro by providing the macro body following the macrodeclaration:Macro DoSomething(real p, matrix|quaternion q, r=) { . . . }

Here parameters q and r are declared as polymorphic: q may be connectedto matrix- or quaternion-valued outputs, while r may be connected to anoutput of any type, provided that type is compatible with the usage of rwithin the macro body. A macro may not be used in code before it isdefined. Forward declarations of macros are not allowed.

In the graph editor, macros appear as encapsulated nodes. The outputsand inputs of a macro node correspond to specific outputs andunconnected inputs of nodes within the macro, so-called public outputsand inputs. These public tabs are declared by identifying particularinputs and outputs as “public” within a node declaration or by issuingexport statements. Exported tabs can be renamed via the “export as”construct. For example: Macro ColoredCubeAndSphere { Color (public r=,public g=, public b=); Group g { Cube (1,1,1); Sphere s; } exports.radius as sphereRadius; export g.ctm_in, g.ctm_out; }creates a macro with r, g, b, sphereRadius and ctm_in inputs and actm_out output.

The first output of a node may exported by declaring the node itself aspublic. In the above example, we could have declared group g as “publicGroup g . . . ” rather than explicitly exporting g.ctm_out.

2.1.7 Naming Conventions

We may refer to node and node-component names by absolute or relativepath. Relative path names begin with “.” or “ . .”, to distinguish themfrom expressions involving division. Model A { real x = 4; Part B { realx = 3; real y = ../x; /* refers to /A/x */ real z = x; /* refers to/A/B/x */ } Scale(x,1,1); /* refers to /A/x */ Translate(./B/x,0,0); /*refers to /A/B/x */ } Model C { Translate(/A/B/y,0,0);/* inter-modelreference, error if A is not loaded */ }2.1.8 Conditional and Iterative Unrolling

Conditional unrolling of model code may be specified via thepreprocessor directives #if, #elif, #else and #endif. These conditionswill be evaluated, and selected branches followed, exactly once, atunroll time (i.e., during initial parsing).

Iterative for and while loops are unrolled at graph construction time,thus these routines do not themselves give rise to iterative orrecursive graph structures. For example: for (i = 1; i <= 5; i++) Avarmake_symbol(”v” + itoa(i));creates five Avar nodes named “v1”, “v2”, “v3”, “v4” and “v5”. Note theuse of the Lisp-like make_symbol to generate symbol names procedurally.These generated symbols may be used like any other name, for example, wecould now write “Scale s(v1,v2,v3);”.2.1.9 Conditional Execution

Conditional graph evaluation may be introduced through conditionalexpressions (i.e., those containing “cond ? a:b”) or by instantiatinggraph structures involving conditional flow (i.e., switches, attachesand IK solvers, as described in Section 1.17). Switches may implicitlybe created through the if-then-else language construct, for example: if(drawSphere == 1) Sphere (5); else Cube (3,3,3);

which is equivalent to: Switch _if0 (nil, drawSphere == 1); Cube(_if0.out_0,3,3,3); Sphere (_if0.out_1,5);where the name “_if0” is generated automatically. The conditional“drawSphere ==1” evaluates to one when true and zero when false. Thecorresponding expression graph is:

The underlined “value” input tab indicates it has an input expression(“drawSphere==1”). A multi-way switch is built by adding additionaloutputs to the Switch node and providing a multi-valued input expression(possibly via a separate Compute node).

2.1.10 Iteration and Recursion

Iterative and recursive graph structures may be created by connectingnode outputs to inputs in order to form a cyclic subgraph. For example:/* create Compute node with one input ”x”, expressing the function ”out= x” */ Compute Fibonacci(real x) = x; /* connect Fibonacci output toits own ”x” input using the given input expression */ Fibonacci.x =t==0 ? 0 : t==1 ? 1 : Fibonacci(t−2) + Fibonacci(t−1).

Thus “Fibonacci(3)” yields the value 2, “Fibonacci(4)” yields 3, and soforth. Referring simply to “Fibonacci” with no explicit parameter yieldsthe Fibonacci number for the implicitly pulled parameter, which only hasmeaning in the context of the surrounding graph. The scoping rules ofthe language are such that a node is declared as soon as its name hasbeen provided, even though the declaration statement has not yet beenclosed by a trailing semicolon. Thus, we could express Fibonacci simplyas: real Fibonacci = t==0 ? 0 : t==1 ? 1 :Fibonacci(t−2) +Fibonacci(t−1).

Here we provide the function as the body of the Compute rather than asan input expression. Note that although Fibonacci is declared as a real,it can be called as a function of (param,lod) and in fact it will returndifferent values for different parameters because its expression dependson the Pull parameter t. As mentioned above, all outputs may be used asfunctions, but they will only return a value that varies with thefunction parameter if the variable definition references t, eitherdirectly, or indirectly through a time- (or more generally, parameter-)varying input. The graph generated by the above statement is shown inFIG. 6.

As mentioned previously, in part due to its ability to represent graphstructures involving iteration and recursion, the modeling language, andthe graph evaluation system itself, is Turing-complete, allowingspecification of any computation or algorithm that could be expressed ina general purpose programming language such as C.

Listing 1 below gives a more complex example of modeling language usage.Listing 1 describes building a desk lamp. Listing 1 #includeonce“global/lighting.m” Macro StandardModelTransforms {  Avar tx=0, ty=0,tz=0;  Avar sx (policy=“*”)=1, sy (policy=“*”)=1, sz (policy=“*”)=1; Avar<quaternion> rot;  Translation trans (tx,ty,tz);  Scaling scale(sx,sy,sz);  public matrix xf_out = trans * rot * scale; } Model Lamp () {  StandardModelTransforms xf;  Joint rootJoint (xf);  Avar r=1, g=0,b=0;  Material metal (diffuse = (r,g,b), specular = (r,g,b), shine =50);  Part Base {   Mesh mesh (“Base.obj”);   Surf baseSurface (mesh,metal);  }  Part Arm {   Joint j0 {    Bone b0 {     Joint j1 {     Bone b1 {       Part Head       {        Avar kind (comment=“0 =pointlight, 1 = spotlight”) = 1;        Avar radius=0.25;        Avarr=1, g=1, b=1, intensity=30;        real c[3] = (intensity * r,intensity * g, intensity * b);        if (kind == 1) {         AvarconeAngle (units=“degrees”) = 30;         Cone (radius*2, radius*1.5);/* draw housing */         Spotlight (radius, coneAngle, color=c);       } else         Pointlight (color=c);        Color (r,g,b);       Sphere (radius); /* draw bulb */       }      }     }    }   }  Mesh mesh (“Lamp.obj”);   SkeletalDeformer deform (mesh, j0);  Surface armSurface (deform, metal);  } }2.2 Expression Language

The language we provide for entering mathematical expressions into nodeinputs uses standard infix mathematical notation. The input will take onthe value of the expression after suitable variable substitutions. Noassignment operator (“=”) is necessary (or allowed) within theexpression. Local-node inputs and global broadcasts (Section 1.16) maybe referenced by name, and there are a number of pre-defined global andlocal variables, constants and a rich set of functions, most of whichare listed in Table 1 below. Expressions are parsed and evaluated by anpublic off-the-shelf POEM (Parse Once, Evaluate Many) parser engine thatapplies aggressive optimization to the parsed expressions, providingfast evaluation speed approaching that of compiled code. TABLE 1 ExamplePredefined Globals, Locals, Constants and Functions. Name Value GlobalsTime global animation clock starttime shot start time endtime shot endtime fps shot frames per second systemdate, systemtime wall-clock dateand time <bcast_name> or bcast[<name>] named broadcast $SHOT, $SEG,$PROJ, $USER, $DEPT, program/system environment $HOST vars Predefinedname local node name Locals path absolute path to local node<input_name> or in[<name>] named input to local node in[i] i-th input tolocal node (zero- based) numInputs number of input tabs param or tpulled parameter value lod.geom, lod.artic, lod.light, . . . effectivelevels-of-detail priority, cost local performance params (see text) ctmconcatenated transformation matrix (only defined for scene graph nodes)Constants pi π = 3.14159 . . . e e = 2.71828 . . . <type>::zerotype-specific zero value <type>::identity type-specific identity valueOperators +, −, *, /, %, {circumflex over ( )}, ˜, !, &, |, <,<=, >, >=, ==, standard infix operators !=, &&, ∥, <<, >> Functions exp,log, log10, log(b,x) exponentials and logarithms sin, cos, tan, sec,csc, cot, asin, acos, atan, trigonometric and hyberbolic atan2,transcendental functions asec, acsc, acot, sinh, cosh, tanh, sech, csch,coth, asinh, acosh, atanh, asech, acsch, acoth sqrt, pow(x,y) squareroot and power min, max, floor, ceil, round, abs, sign, numericalfunctions fmod srand, rand pseudorandom numbers noise1D, noise2D,noise3D Perlin noise binomial(n,m) binomial coefficient lerp, bilerp,slerp, smoothstep interpolation cond ? a:b binary conditional<expr>.Derivative(n) n-th time-derivative of given input<expr>.PartialDerivative(n,v) n-th derivative with respect to v<expr>.Integral(a,b) definite integral over interval [a,b]

In addition to what is shown in the table, we provide functions forbasic character and string manipulations; operators for array, mesh andmatrix indexing; basic matrix and quaternion functions and operators,and so forth.

2.3 Animation Language

In Section 1.18.1, we described our use of avars, or articulatedvariables, to provide time- (or more generally, parameter-) varyingvalues within our expression graph. In Section 5, we describe thelayered representation of avar data and detail the process by whichlayers are combined to produce an articulated output value. We alsodescribe, in Section 5.2, the taxonomy of avar motion channels,identifying kinematic channels as those involving the splined animationtypically created by a human animator. Because of its suitability tointeractive specification and modification, kinematic animation is inmany ways the most important layer class within our system. Here wedescribe the text-based language used to specify kinematic data. Otherlayer classes are specified via tab-delimited tables (sampled layers),function pointers (procedural layers), parameters values (stochasticlayers), or through the combination of parameter values and boundaryvalues specified in lower-numbered motion channels (forward-dynamic andmultipoint-dynamic layers).

Following common practice, our kinematic data consists of knots alongparameterized one-dimensional curves or splines: each kinematic motionchannel has one spline. Each knot specifies the value the spline takesat a particular parameter and possibly some additional information aboutthe curve shape to either side of the knot. The spline is interpolatedbetween knots. The data stored at a knot, and the algorithm used forinterpolation between knots, depends on the knot type. We provideseveral knot types commonly used in prior systems, including stepped,linear, Bezier, Hermite, b-spline, cardinal, TCB and tau. For Bezier andHermite knots, we provide left- and right-tangent handles that indicatethe left and right derivatives at the knot. Interpolating a splinesegment between knots of dissimilar type involves a two-stepinterpolation: first we interpolate the segment twice, once as eachspline type; then we interpolate between these two splines.

The information we provide in specifying channel data includes: the pathto the avar, the layer in which the channel resides, the curve parameterat which each knot occurs, the value of each knot and anyknot-type-specific parameters (per knot). The parameters for each knottype are given in Table 2 below. We may optionally also specify thelayer set (Section 5.4) to which the layer belongs. TABLE 2 Knot Typesand Associated Parameters Knot Type Code Parameter List Stepped D emptyLinear L empty Bezier B Left- and right-handle length and derivativeHermite H Left- and right-handle length and derivative B-spline S emptyCardinal C tension (tension = 0.5 yields Catmull-Rom) TCB T tension,continuity, bias (Kochanek-Bartels) Tau U tension, continuity

Statements within an animation file define the scope of a particularavar, select which layer is being referenced, or provide the actual knotdata. The format is a subset of that described in Section 2.1, with theadditions of a “Layer layername” statement for scoping by layer and knotstatements of the form “u: knot-code value parameter-list”. Layers andscene hierarchy are scoped independently, that is, “Layer C Model M{Avar tx { . . . }}}” is equivalent to “Model M {Layer C {Avar tx { . .. }}}”, “Model M {Avar tx {Layer C { . . . }}}”, “Avar/M/tx {Layer C { .. . }}”, and so forth. An example kinematic animation data file is shownbelow in Listing 2. More particularly, Listing 2 is for kinematic motionchannels for five avars; “/Greg/LArm/wristRot” is a quaternion-valuedavar (with an extra “spin” parameter at the end of each knot statement),the others are scalar-valued. Listing 2 /* Reflex cuesheet version 0.3 * Auto-generated by Reflex dev build Jul 16 2003 18:23:58  * User mjoon 7/21/03 at 12:20 PM  *  * File: /Users/mjo/shots/Dive.cue  */ Layer“Up-down arc” {  Model Greg {   Part LArm {    Avar ikGoal_tx {     30:B −0.0203 0.05 0.05 0 0     47: L 0.0142     56: L 0.2506    }    AvarikGoal_ty {     30: B −0.0275 0.05 0.0801 0.12 0.12     47: B 0.42190.082 0.3912 0.1511 0.1511     56: B 0.1812 0.151 0.151 0.0021 0.0021   }    Avar ikGoal_tz {     30: B 0.1243 0.05 0.05 0.91 0.91     47: B0.0903 0.11 0.11 0.1491 .125     56: B 0.113 0.051 0.051 0.08 0.08    }   Avar wristRot {     12: B (0.623, 0.681, 0.2, 0.29) (0,0,0,1)(0,0,0,1) 0.1, 0.05, 0     37: B (0.7799, 0.54, 0.253, 0.18) (0,0,0,1)(0,0,0,1) 0.05 0.01 0     84: L (0.578, 0.6356, 0.4459, 0.2512) 0    }  }   Avar tz {    43: B 0 0.05 0.05 0 0    55: B 2.3851 0.4622 0.3395 00    63: B 0.4347 0.01 0.05 0 0    64: B 0.5303 0.05 0.05 0 0    66: B0.1788 0.05 0.05 0 0    71: B 0.3649 0.05 0.05 0 0   }  } } /* END OFCUE DATA */

As mentioned in Section 2.1.3, the animation specification syntaxdescribed above is not properly a distinct language from the modelinglanguage of Section 2.1. In fact, animation may be specified within oralongside general model code by scoping layers and associated knot listswithin Avar node declarations, as shown in Listing 3 below. Listing 3#includeonce “std_defs.m” Model Bottle {   StandardModelTransforms xf;  Joint rootJoint (xf);   /* Specify bottle radii at various u's: u=0 atbottom, u=1 at top.    * Note this restricts us to a shape that does notbend back on    * itself. A better shape could be had by using apoint-valued    * avar and from that driving the Revolve node directly.   *    * We pass param=nil so the avar is not time-driven.    */   Avarradii (param=nil) {     Layer “Construction” {       /* u: knot-typevalue params */       0.000: B 0.00 0.05 0.05 0.0 0.0       0.010: B3.97 0.05 0.05 0.0 0.0       0.070: B 4.00 0.05 0.05 0.0 0.0      0.400: B 4.00 0.05 0.05 0.0 0.0       0.750: B 2.50 0.05 0.05 0.00.0       0.930: B 2.20 0.05 0.05 0.0 0.0       0.940: B 2.25 0.05 0.050.0 0.0       0.990: B 2.25 0.05 0.05 0.0 0.0       1.000: B 0.00 0.050.05 0.0 0.0     }   }   /* Build mesh surface-of-revolution. */   realheight=10;   Curve3D profile (umin=0, umax=1, x=radii, y=u*height, z=0);  Revolve revolve (profile, ‘y’);   /* Generate surface from mesh. */  Material (“glass”);   Surface (revolve); }3. Ghosts

In addition to the normal rendering of geometry described in Section1.10, we optionally draw geometry as it would be posed at alternatetimes t, or with alternate animation variable settings, translucentlyinto the scene as ghosted geometry. Normally only the “active” model(the model containing the active node) is drawn ghosted, however theuser may override this by specifying particular geometry to include inthe ghosted renders.

The user may specify one or more arbitrary times at which to draw aghost. Ghosts may be drawn at regular intervals in time, for example:“on 24's”, that is, once-per-second for a 24-frames-per-second (fps)animation, as shown in FIG. 8 b. Ghosts may be drawn at “knots”, orkeyframed values, in the animation (Section 4, FIG. 8 a). Ghosts may bedrawn at designated “key poses” as defined by user-specified timemarkers associated with sections of geometry. Ghosts may be drawn at aparticular motion path knot during interactive manipulation of that knot(Section 4, FIG. 8 c). In some implementations, the user may set apreference to have such manipulation ghosts shown (1) never, (2)momentarily during interactive manipulation, or (3) always when one ormore path knots are selected. Other preferences are possible. Severalconsecutive ghosts may be drawn during adjustment of knot interpolationrates, such as while the user is dragging frame-timing beads along themotion path or manipulating path knot tangent handles (Section 4.2.4).Rendering ghosts at alternative times is accomplished by performing aRender pass over the specified nodes after (1) setting the globalanimation clock to the ghost time, and (2) rendering the ghosts using anon-opaque color (alpha <1.0) with OpenGL or equivalent blending.

Ghosts may be shown to assist in comparing alternative animation,specifically with differing sets of enabled layers (Section 5). We cando this by rendering the scene once normally, then adjusting whichlayers are enabled, and then rendering the ghosted nodes a second time.Because changing enabled layers invalidates the affected portion of ourexpression graph, the ghosted nodes will be rendered under thealternative layer setup.

User controls over ghost rendering include: ghost opacity (“alpha”),illumination mode (whether to perform per-vertex lighting computationsor use constant shading), fade rate (ghosts may fade in opacity awayfrom the current time, as shown in FIG. 8 b), and draw style (ghosts maybe drawn as surfaces or “stick-figure” skeletons). A special “Dopplershift” mode colors ghosts at times earlier than the current timedifferently than ghosts later than the current time (blue-tinted andred-tinted, respectively, in our implementation). Finally, the user maychoose whether or not ghosted geometry is selectable via the mousecursor. By default it is not selectable, and clicks on ghosts have thesame effect as they would if the ghost was not there, for example, theywill select opaque geometry drawn behind the ghost.

Ghost rendering follows any distortions of the motion path as describedin Section 4.1.2 and shown in FIGS. 9 a-9 c. This is accomplished simplyby pre-multiplying the ghost concatenated transform matrix (CTM) by thepath distortion transforms in Section 4. Our ability to rapidly renderghosts across multiple animation times and layer settings is enabled bythe temporal caching scheme described in Section 1.

4. Motion Paths

In this Section, we describe the use of motion paths for visualizing andcontrolling the movement of articulated and deformable bodies. A numberof animation systems today are able to display a curve through 3D spaceindicating the path followed by an object moving through that space. Werefer to such a space curve as a motion path. Some of these systems alsoprovide user control over the motion path by allowing the user todirectly manipulate the shape of the path, for example, via controlpoints on the path. This effectively lets the user specify translation,and possibly rotation, of an entire object traveling through space bymanipulating the path of the object directly. Unlike previous systems,we extend this motion path representation by providing control ofarticulated bodies and deformable surfaces, allowing not only the pathof the whole object to be viewed and manipulated, but allowing changesin the shape of the object itself to be viewed and manipulated (FIGS. 8a-8 c and 9 a-9 c).

Some notable features of our technique are that it is very fast due totemporal caching; it is flexible enough to control any type of motion,including rotation, translation, and scaling of and within articulatedbodies, surface deformations, and global deformations; and the referencepoint through which the motion path is drawn, and to which refers usermanipulation of the path, may be any point on or relative to thesurface.

4.1 Visualization

4.1.1 Path Shape

Our technique for implementing motion paths relies on temporal cachingwithin our expression graph (refer to Section 1). A user may specify anyreference point on any surface in the 3D scene by a number of means (forexample, by clicking directly on that surface while depressing adesignated key on the keyboard). To draw a motion path indicating wherethat reference point on the surface moves through space over time, wecompute where in world-space (i.e. global) coordinates that point isacross some series of discreet sample times, and connect these pointswith line segments. We render these line segments as a connectedpolyline, giving the appearance of a curve through space. Note that wedefine “curve” herein to include straight line segments. Varying thespacing between time samples allows us to produce a smooth curve withoutunnecessary detail; the adaptive method for computing this spacing isdescribed below. What remains is to describe how we compute the 3Dposition of the reference point at any given time. For simplicity wedescribe this assuming the reference point lies on the surface; notehowever that the reference point could as easily lie off the surface:all that may be required is that we know its position relative to thelocal surface coordinate system.

The position of any point on a surface is governed by the cumulativeeffect of a series of mathematical operations defined by our expressiongraph. We distinguish here two classes of cumulative effect: (1) rigidand (2) non-rigid (refer to Section 1.10). In the first case ofconcatenated rigid transformations, the position of the reference pointP_(ref) may be represented as a 3D point P_(local) in the pre-transform(local or body) coordinate system of the surface containing that point.The local coordinate system itself is defined by the 4×4 ConcatenatedTransformation Matrix (CTM) C_(surf) with which the surface is drawn.Computing the reference point position at any time t involvestransforming P_(local) by the value of C_(surf) at t. C_(surf)(t) isavailable to the surface node at its CTM input (refer to Sections 1.9and 1.18). We automatically enable temporal caching of the CTM for asurface through which we are drawing a motion path (refer to Section1.5). Thus we approximate the world-space position at time t of areference point lying on a surface as:P _(ref)(t)=C _(surf)(t)P _(local)≈ψ_(M(t)) P _(local)where ψ is the cache lookup operator for the expression node outputconnected to the CTM input of the surface.

In the second case of point motion governed by non-rigid deformations,we are faced with one of three alternatives. First, we may regard thedeformations as being “nearly rigid”, and compute the point motion asabove. Second, if we desire an accurate representation of the motionunder a surface deformation, we apply the deformation to the referencepoints. Let D_(surf) be the time-varying function that maps points totheir position under the surface deformation, then:P _(ref)(t)=C _(surf)(t)D _(surf)(t,P _(local))≈ψ_(M(t)) D _(surf)(t,P_(local)).

Finally, if we represent the motion of a point under a global, ratherthan surface-relative, deformation, we pass transformed points throughthe corresponding D_(global) function:P _(ref)(t)=D _(global)(t,C _(surf)(t)P _(local))≈D _(global)(t, ψ_(M(t))P_(local)).

For complex motion involving a mixture of rigid transformation, surfacedeformation and global deformation, we may combine these operations asneeded.

The motion path polyline itself is represented in our expression graphas a MotionPath node (a subclass of the Glyph node type, see Section1.18), which has among its inputs the object or surface to which thereference point belongs as well as the surface-relative position of thatpoint. Thus we only recompute the motion path as necessary by changes inthe reference point position (due to the user specifying a new referencepoint) or by changes to the transformations and deformations affectingthe surface (due to user changes to the underlying animation).

4.1.2 Path Display Alternatives

In complex scenes the motion path can become difficult to see andunderstand clearly. This is particularly true when the path is tightlybunched: it may begin to resemble a ball of yarn. We have severaloptions available for improving the display of the path in suchcircumstances.

When fine detail of the path is difficult to discern, the user mayenable path dilation (also referred to as path magnification). Thisenlarges, or dilates, the path about some reference point in space. Weaccomplish this by translating the points of the path polyline so thedilation center sits at the origin, scaling the points by auser-specified dilation factor, and then translating them back:P _(dilated)(t)=T ⁻¹ STC _(surf)(t)P _(local) ≈T ⁻¹ STψ _(M(t)) P_(local)where T is the transformation that translates the dilation center to theorigin, and S scales uniformly by the dilation factor. By default wewish the path to still pass through the target surface at the currentframe, for which we provide a default mode in which the path targetpoint is the dilation center. In this mode, when we play the animationand the target point moves in world space, the path will move with itsince the dilation remains centered about the target point.

When the path becomes difficult to follow because it doubles back overitself repeatedly (the ball-of-yarn problem), the user may enable thepath tickertape option (see FIGS. 9 a-9 c and 10 a-10 b). This has theeffect of stretching the motion path out in time within the 3D worldspace by mapping time to a 3D vector in that space. The direction of thevector controls the direction in which we stretch the path, and themagnitude of the vector controls how much the path is stretched. To drawthe path in this mode, we translate each point in world-space by thetickertaping vector V scaled by an amount that increases with t:P _(tickertape)(t)=(t−t _(now))V+C _(surf)(t)P _(local)≈(t−t _(now))V+ψ_(M(t)) P _(local).

Note that the path sample point at t=t_(now) is unaffected bytickertaping; therefore the path continues to pass through the targetsurface/target point at the current frame, and when the animation isplayed the path will seemingly flow through space in the directionopposite the tickertaping vector so as to keep the path correctlypositioned on a moving target surface (FIGS. 10 a and 10 b).

FIGS. 7 a-7 c are screenshots illustrating manipulators for controllingdilation and tickertape. Sometimes motion higher up an articulatedhierarchy makes it difficult to visualize movement at the level in whichwe are interested. For example, we wish to view the path of the left armswing of a character as the character walks across the screen. The pathwill travel across the screen due to the motion of the character;however, we may wish to see the path of the arm swing in isolation,without the effect of the cross-screen movement. We allow the user tospecify any reference node relative to which to draw the path. In thisexample, the user might choose to draw the path relative to the leftshoulder joint or to the root joint (e.g., the hips) of the character.We implement this by transforming path sample points into the coordinatespace of the reference node before plotting the path polyline:P _(relative)(t)=C _(reference) ⁻¹(t)C _(surf)(t)P_(local)≈ψ_(reference,M(t)) ⁻¹ψ_(surf,M(t)) P _(local)in the case of rigid-body transformations, and we make an analogousextension for the non-rigid cases. Here we will cache the inverse CTM ofthe reference surface C_(reference) ⁻¹, and ψ_(reference) ⁻¹ is thecorresponding cache lookup operator. Note that this does not limit us toreference nodes in the scene hierarchy above the motion path targetsurface; the path may be drawn relative to any node in the 3D scene. Ifwe specify the path be drawn relative to the target surface itself, thedisplayed path will show the motion due only to deformations of thatsurface.

One particularly useful application of this is to draw the path relativeto the camera when the camera we are looking through is moving. Thiscauses the path to indicate how the target point moves within the cameraview, rather than through world space. This is useful both for animated“shot” cameras and for attached “detail” cameras that move with thetarget surface, and we provide hotkeys for enabling these modes.

To improve visibility of the path over different-colored backgrounds,and to help in distinguishing among multiple paths, the user may specifya color for the path. The user may also choose to have the path drawn intwo alternating colors (colorstriping), with the color changing oversome interval of t (time-striping, FIG. 8 b), or at every knot (seeSection 4.1.3 below), or at user specified markers (also referred to asposes). The user may alternatively choose to enable the Doppler shiftmode, where the section of the path over samples preceding t_(now) isdrawn in one color and the section following t_(now) is drawn in adifferent color. Or the user may elect to have the path fade out atsample points progressively farther in time from t_(now) in the same wayghosts may be faded in and out through time (Section 3, FIG. 8 b).

Sometimes it may be difficult to see the path because it is obscured byforeground geometry. The path bleed-through mode causes the pathpolyline to show through obscuring objects as though they weretranslucent (FIG. 8 a). We accomplish this by rendering the path twice,first fully opaque (that is, with “alpha” or opacity=1.0) with depthculling enabled, then a second time translucently (that is, with auser-specified opacity <1.0) after all other scene geometry has beendrawn.

4.1.3 Path Decorations

In addition to the polyline of the path itself, we display several otherpieces of information along the motion path. At sample points that fallon integral (whole number) frames, we optionally draw a small blue markor bead (FIG. 8 a). This gives an indication of the rate of movementover a section of the path: where frame beads are drawn close together,the motion will be slow; where the beads are far apart, the motion willbe fast. The user can click and drag with the mouse cursor on thesebeads to adjust interpolation rates (Section 4.2). This providesvisualization and control over the traditional animation concepts ofease-in and ease-out.

The user may click on frame beads or drag a selection box around thosebeads with the mouse cursor to select parameter values, for example, toselect the corresponding shot times when the avar input parameter t istime. Subsequent program commands involving time will operate on theselected times. Selecting two or more contiguous frame beads selects acontinuous parameter range. For example, the user may select a timerange and then choose Copy from the application menus; this will copyanimation of the active nodes within the selected time range. Thisanimation may be subsequently pasted over a different time range or on adifferent group of geometry.

At sample points that fall on times at which active avars have knots(Sections 1.18.1 and 2.3) we draw a yellow diamond shape (FIG. 8 a).This indicates the timing and value of animation keys. Timing isindicated both by the position of the knot along the path and explicitlyby displaying next to the knot the frame number at which the knotoccurs. Value is indicated by the position of the knot in 3D space and,for selected knots only, optionally explicitly by displaying next to theknot the numerical value(s) specified in the associated avar(s). Onlyknots for unlocked degrees of freedom within the model are shown. Forexample, when performing an axis-constrained manipulation (e.g.,dragging on an axis-constraint handle of a manipulator widget or holdingdown an axis-constraint hotkey), only knots for the unconstrained avarsare displayed on the motion path. The user can click and drag with themouse cursor on a knot to adjust the knot value or timing (Section 4.2).We optionally display a text label next to the motion path knotindicating the names of the underlying avars possessing knots at thatknot time (along with numerical values for those avars, as describedabove).

Knots that result in changes to the effective graph topology (forexample, knots that enable or disable inverse kinematics, attaches, orhierarchy inversions, or that change graph switch-node settings) displayon the motion path as a yellow diamond with a yellow line outline.

The user may select a knot on the motion path (e.g., by clicking on theknot with the mouse cursor or by dragging a selection rectangle over theknot). Selected motion-path knots that represent non-linear knots in theunderlying avars will display path-tangency handles. The length anddirection of these handles reflect the tangent-handle settings for theunderlying avar knots, and direct-manipulating these path handles willmodify the avar knot handles and thereby the motion path shape (Section4.2). We compute the length and direction of the motion path handles bydifferentiating the motion path curve at the knot (Section 1.14). Whenone or more of the underlying avar knots has broken tangent handles(Sections 1.19.1 and 2.3), we draw the motion path knot as an orange,rather than yellow, diamond shape (FIGS. 8 a-8 c), and the path knothandles will themselves be drawn and behave in a “broken” fashion, thatis, the left and right handles will be independent.

When tangent handles are drawn, we optionally display a rotation twistindicator on the local-coordinate-system +Z axis-line extending to theknot that indicates the direction of a perpendicular axis (e.g., +X or+Y); this allows visualization of how the articulation twists along itscentral (i.e., Z-) axis. The twist indicator can be used as adirect-manipulator to control interpolation of axial twist betweenmotion path knots (Section 4.2.4).

On the tangent handles themselves we optionally display a slidinghashmark that indicates the rate at which the motion approaches (for theleft handle) and departs (right handle) the path knot. The farther themarker is up the handle away from the knot, the faster the motion willmove; these marks can be direct-manipulated via the mouse cursor toslide them up and down the handle and thereby control the motion rate(Section 4.2). This provides interpolation-rate visualization andcontrol of ease-in and ease-out analogous to that provided by theframe-bead spacing described above. The user is free to have multiplepaths drawn, each through a different piece of geometry or through adifferent target point on the same geometry. These multiple paths may bedistinguished by setting each to a different color and/or by displayingtextual annotations alongside each path that indicate the target surfacename, the controlled bone or joint name, or a user-specified motion-pathname.

FIG. 8 a is a screenshot of a user interface 800 for an exemplary 3D CGanimation system. In the example shown, the user interface 800 includesa view window 801 for presenting a display environment containing ananimated body 802. For this particular example, the animated body 802 isa bendable-bottle model. Other bodies are possible. The term “body”includes but is not limited to: rigid objects, models, wire structures,articulated bodies, soft bodies, bodies with deformable surfaces, etc.

A motion path 804 is displayed in the display environment 801 with knots806 and frame beads 808. The motion path 804 includes knots 806 for theactive avar “rot_2” at frames 0, 15, 30, 62, and 98. Motion tangents arebroken at frame 15. The knot 806 b at frame 30 is selected and its(continuous, non-broken) handles 807 displayed. Ghosts 810 are drawn ateach of these keyframes. We can see motion path bleed-through has beenenabled, as the motion path 804 translucently shows through where itpasses behind the bottle surface between frames 87 and 95. We see markerlines 813 a, . . . , 813 d at the keyframes in the timeline 809 belowthe view window 801, as well as a selected line 815 for the knot atframe 30 and a current-frame marker 817 at frame five; these lines arealso drawn on the scrollbar 812 immediately below the timeline 809. Therange bar 811 below the scrollbar 812 sets the time range for which themotion path 804 will be drawn.

FIG. 8 b is a screenshot of another bottle animation showing time-fadingghosts at 1-second intervals and motion path colorstriping. In theexample shown, the active avar “rot_2” has knots at frames 0, 54, 71,and 94. The knot 819 at frame 54 has broken tangents. Frame beads arehidden and colorstriping has been enabled. The colorstriping interval onthe motion path 821 matches the colorstriping interval on the timeline809 below the view window 801. Ghosts 814 a, . . . , 814 e, are drawn atone-second intervals; we can see their exact timing indicated by markerlines 817 a, . . . ,817 e, in the timeline 809 and timeline scrollbar812. Ghost fading is enabled, with ghosts 814 a, . . . ,814 e, fadingout away from the current frame eight.

FIG. 8 c is a screenshot of another bottle animation showing amanipulation ghost 822 with an arcball manipulator 816. In the exampleshown, the user has selected the knot 820 at frame 94. The absence oftangent handles indicates this is a linear knot (or almost-linear knotwith very short tangent handles). The manipulation ghost 822 has beencreated automatically at frame 94. The box 823 around the active avarname “rot_2” confirms there is a knot at the selected time. The round“arcball” manipulator 816 has been displayed around the “Bottle” model's“joint_2” node; it is drawn with its Y-axis constraint-ring highlightedto indicate the user is performing axis-constrained rotation about thejoint-local Y-axis, perhaps because the user is holding down anaxis-constraint hotkey or has initiated dragging by clicking directly onthe Y-axis constraint-ring. The even spacing of frame beads 824 alongthe entire motion path 818 indicates that the rate of motion along themotion path 818 will be uniform, with no ease-in or ease-out.

FIGS. 9 a-9 c show a character 902 walking in-place (he does not haveany forward motion through space). In FIG. 9 a, we observe the motionpath 902 of the end of the nose: it is an up-and-down cycle thatoverlies itself. A ghost 904 is shown at every keyframe. In FIG. 9 b,the user has enabled motion path tickertaping: now the motion path 902is stretched sideways and we can visualize the complete movement cycle.The ghosts 904 are displaced in space along with the correspondingsample points on the motion path 902. In FIG. 9 c, the user has orbitedthe camera to look from the front, and has swung the tickertapedirection to keep the motion path 902 stretched out to the right side;she has also increased the magnitude of the tickertape vector so themotion path 902 is more stretched out. The ghosts 904 remain correctlypositioned relative to the path knots 906.

In FIG. 10 a, the user has selected the root joint 1002 of the model1000 and has tickertaping enabled. The bones of the underlying skeleton1004 are translucently overlaid over the model surface 1006. At framezero, we can see the motion path sample point (which happens to be at aknot) at frame zero lies at the root joint 1002; the rest of the motionpath 1001 stretches off in the tickertaping direction. In FIG. 10 b, theuser has advanced to frame nine; now the motion path 1001 has shiftedlaterally so that frame 9 on the motion path 1001 lies at the root joint1002.

FIG. 11 is a flow diagram of an exemplary motion path manipulationprocess 1100. In some implementations, the process 1100 begins byproviding a display environment for presentation on a display device(1102). The display environment can be a 2D or 3D virtual environmentprovided in a view window of a user interface that includes a scenehaving one or more graphical objects. A body or model is provided fordisplay in the display environment (1104). The body can be a rigid body,an articulated body, a body with one or more deformable surfaces, or anycombination thereof. An example of an articulated body is the walkingcharacter shown in FIGS. 9 a-9 c and 10 a-10 b. An example of a softbody is the bendable-bottle shown in FIGS. 8 a-8 c. A motion path candefined for a first portion of the body relative to a second portion ofthe body or relative to a coordinate system associated with the displayenvironment (1106). A control is provided in the display environment formanipulating the motion path (1108). The control can be a knot, asdescribed with respect to FIG. 8 a. Other controls are possible. Thepath can be manipulated in response to a user interaction with thecontrol (1110). For example, the user can use knots to change thestarting and end positions of the motion path, change the shape of themotion path, etc. Other motion path manipulations have been previouslydescribed with respect to other figs.

4.1.4 Adaptive Path Resolution and Range

From the above description of path rendering, it should be clear thatoptimally we want to plot the position of the path only for values of tthat cause valid cache mappings; in general we pick t's that fallcentered within our cache lines. However, in some circumstances we maydesire finer or coarser temporal resolution than the cache linesprovide.

When an object is moving very slowly in space, the path will tend tochange little between subsequent sample points. Rather than drawing apolyline through every sample point, we skip points that vary little inscreen space from neighboring samples. We do this locally in t, startingwith a coarse path and refining segments of the path whose midpointsdiffer from a straight-line interpolation of the segment endpoints bygreater than some screen-space error threshold. We provide user controlover the error threshold: selecting a high threshold causes the path tobe less accurate but render faster.

When an object is moving very quickly through space, the path will tendto become stretched out. In such cases, sampling P_(ref)(t) at eachcache line may not provide an accurate representation of object motion.For the greater cost of evaluating our expression graph at non-cachedt's, we may provide finer-grained sampling of the motion. We provide auser option to enable “accurate paths”; under this option we use ascreen-space error metric to determine whether and how far we shouldlocally refine any segment of the path beyond the cache-sampling rate.

We optionally scale the path subdivision threshold e_(thresh) by themotion path Level of Detail lod, so that the path becomes less (or more)accurate as the LOD of the path decreases (or increases) (see Section1.7).

Because rendering a very long path (i.e., one that extends across a widerange of t) can be slow, we optionally increase the error threshold as tdiverges from the global animation clock t_(now):e′ _(thresh)(t)=f(t−t)*lod*e_(thresh)giving us a time-varying threshold e′_(thresh)(t), where f(x)=1 at x=0and f(x) increases as the magnitude of x increases. We have found thatsimply: ${f(x)} = {1 + \frac{c{x}}{t_{\max} - t_{\min}}}$

often works well, with c providing linear control over the rate ofaccuracy degradation away from t_(now). This procedure yields a paththat is accurate around the “current time” but becomes progressivelyless accurate farther from the current time. We automatically disablethis option (i.e., set f(x)=1) whenever the global animation clock isdynamically changing, such as when the user plays the animation.

We also allow the user to specify the range of parameter values[t_(min),t_(max)] over which the motion path will be drawn; specifying anarrower interval both makes the motion path render faster and reducesclutter on the screen. The green range-bar seen at the bottom of thetimeline in FIGS. 8 a-8 c provides one means of controlling the pathrange. We optionally also display in the camera view range-manipulatortabs at either end of the motion path itself: dragging on one of thesetabs or selecting a tab and typing in a new time adjusts thecorresponding setting for loop start or end times.

4.1.5 Automatic Path Placement Option

When manipulating an object, we often want a motion path to be drawnthrough that object. In the case of an articulated body consisting ofbones connected by rotational joints, it is it often is convenient for amotion path to be drawn through the end of the manipulated bone. Tofacilitate this, we provide the option of automatic path placement. Withthis option enabled, selecting any scene object causes a motion path tobe created drawn through the root of that object. Selecting a part (suchas a bone) within an articulated object causes a path to be drawnthrough the distal end of that part. Selecting a model (such as acharacter) causes a path to be drawn through the root joint of thatmodel.

4.2 Manipulation

In addition to showing us the path and timing of motion, our motion pathprovides full direct-manipulation control over motion. The user canclick and drag with the mouse cursor on a knot or its handles to adjustknot value, timing, motion tangents, twist rates and interpolationrates.

In general, manipulating motion via the path involves inverting themechanism that is responsible for the motion being what it is, that is,the mathematical and procedural computations that map from changingavars to 3D movement; these computations are exactly those representedby the relevant subsections of our expression graph. This inversion ofthe graph computations allows us to map from desired changes in the pathshape and timing to the necessary changes to be made in the underlyingavars. In some cases the forward mapping from avar to motion will benon-invertible; in such cases a “pseudo-inversion” is readily availablefrom our expression graph, as detailed below.

In addition to knowing what changes need to be made to avars, we alsotrack dependencies arising from this control mechanism. Changes to themotion path cause dependent avars to be adjusted. Adjustment of avarknots has the side effect of pushing dirty on the avar outputs over theinterval of support of the knots. This push will recursively push dirtythrough the articulation and/or deformation hierarchies, ultimatelypushing the active surface. The motion path itself is a child of theactive surface, and so it too will receive the push that it originated.Thus manipulation of the motion path forces us to re-evaluate and redrawthe surface and, by extension, the motion path itself. Thisstraightforward dependency-based reciprocal invalidation of surface andmotion path is enabled by our toleration of cycles within the expressiongraph.

Avar discovery is the process of traversing the graph and enumeratingthe avars up-graph of a particular node. This avar list will define thedegrees-of-freedom that exist in animating that node. Avar discovery isnormally done at model unroll time, that is, when the model is firstinstantiated in core memory from disk at the beginning of an animationsession. However, we can also perform avar discovery on a just-in-timebasis, deferring discovery for every node until the first time that nodeis made active. Static changes in graph topology (for example, creatingor deleting nodes or adding or removing edges) may require that werediscover upstream avars. Dynamic topology changes (due to switches,inverse kinematics state changes, attaches, and hierarchy inversions)may not require rediscovery however; we simply maintain multiple listsfor the upstream avars of affected nodes, and use whichever list iscurrently applicable based on the current topology. This is possiblebecause these dynamic topology changes are discrete changes in which thealternative topologies are fixed and thus enumeratable at unroll time.

4.2.1 Graph Inversion

In Section 1, we detailed how our expression graph represents thecomputations and dependencies underlying our 3D scene. We identifiedthat this is a directed graph, that is, edges in our graph havedirection, and that the direction indicates both the conceptual flow ofinformation and (in the opposite direction) computational dependency. Wealso discussed (in Section 1.17) how we can locally invert a subgraphwithin our expression graph by inverting each node and switching theedge directions. Recalling from that section, nominally it falls uponeach node type to define its own inverse; however, the system canestablish a local inverse (a Jacobian) for nodes that fail to definetheir inverse function. The Jacobian is not a true inverse because itonly is valid around a local region of parameter space.

Here we use this same mechanism to map desired changes in knot positionto changes in the animation control variables (avars). We recall thatthe forward computation defined by the expression graph maps avarsettings to scene configurations. Said another way, the forwardexpression graph maps from avar parameter space to 3D configurationspace. By inverting (or pseudo-inverting) the expression graph, asdetailed in Section 1.17, we can map from 3D configuration space to avarparameter space. Specifically, we can map from local changes inmotion-path knot position to corresponding changes in the underlyingavars. This local inverse mapping is precisely what the Jacobiandefines. For optimization of pseudo-inversion across complex graphstructures involving deformations and similar “expensive” operations, wecompute the composite Jacobian across the subgraph using the method ofSection 1.17. This is convenient when the user is direct-manipulating amotion path dependent upon expensive articulations and deformations.

In certain circumstances, the local pseudo-inversion provided by Section1.17 is not capable of successfully adjusting avar parameters in a waythat would achieve the desired change to motion-path knot position, eventhough such a change is possible. This occurs when the configurationbecomes stuck in local minima from which our local view of the inverseis not able to escape. Such circumstances mostly arise only inpathological graph configurations, so they do not overly concern ushere. When they do arise, the user will rely on other manipulationmethods, such as direct manipulation of skeletal joints and bones oradjustment of the underlying avars themselves, to escape the localminima.

In many cases we can avoid issues of local minima by ensuring our nodesdefine globally valid inverse operators. In some cases this won't bepossible. We can employ an automatic stochastic technique for escapinglocal minima. This is accomplished simply by injecting noise into theavar parameters in the hope that a “lucky” random change to an avar willallow us to escape the local minimum. In practice, generally we havefound it to be more convenient to allow the user to direct-manipulatethe skeleton or other geometry out of the local minimum. Becausemanipulation of the motion path knots, the skeleton or geometry itself,and the actual avars all affect the same underlying representation (theavars), the user may freely move between these control mechanisms, forexample by “untwisting” or otherwise manipulating geometry directly whenthe motion path knots get “stuck”. Nonetheless, we do provide stochasticescape of local minima as a user option.

4.2.2 Knot Value Adjustment

Each motion path knot represents some non-empty set of knots on unlockedactive avars (Section 4.1.3). When the user drags with the mouse-cursoron a motion-path knot, we wish to adjust the corresponding avar knots asnecessary to keep the motion-path knot under the cursor. In general, asdescribed in Section 4.2.1 and elsewhere above, this is anunderconstrained problem (that is, there are multiple “good” solutions),although in some situations it will be overconstrained (there is nosolution). We find a “best” solution as follows.

First, we need to identify a point in 3-space to which we want themotion path knot to move. The mouse position defines a ray in 3-spacethat begins at the camera-space origin and extends through theprojection of the mouse cursor on the screen plane. We choose as ourtarget point for the motion path knot the intersection of this ray withthe plane passing through the current motion path knot position andperpendicular to the camera forward (−Z) axis. Thus the current knotposition and the target knot position both lie in the same X-Y plane.Subtracting the current position from the target position gives us agoal vector in that plane. We now need to find the change to the activeunlocked avars that produces movement in the path reference point thatbest approximates this vector.

In the simplest case we will have an explicitly invertible pathavailable through the expression subgraph that subsumes the active avarsand the path target surface. In this case it is straightforward to solvefor the necessary avar changes by passing our goal vector through theinverted graph, yielding a vector of avar deltas. These deltas areapplied to the avars as described in Section 5.

In another simple case, we do not have an explicitly invertible path butthe composite Jacobian for this subgraph is orthogonal with respect tothe control avars. Here it is straightforward to transform the vectorthrough the Jacobian and extract the necessary avar deltas. These deltasare then applied as described in Section 5. We then examine how closelythe motion path knot approaches the desired position, and iterate untilwe reach a preset error bound. This iteration is independent of framerendering (see Sections 1.6 and 1.8) and therefore does not interferewith maintaining target rendering rates. However, because we do maintainfast update rates, rarely does the mouse move far enough in a singleupdate pass that one or two iterations are insufficient to achieve theerror bound, so we rarely need to iterate further. Because we arerelying on a gradient descent method, we have no problem toleratingoverconstrained systems; we will find a “good” solution, if not theexact solution. We deal with local minima as described in Section 4.2.1above.

In more complex cases, the composite Jacobian will not be orthogonal andmay have degeneracies. The system may still be under- orover-constrained. There are a number of ways to deal with such systems;we take the pragmatic approach of the model specifying preferred avarsettings (as defined by the avar default values) and limits andweighting particular avars as being “more stiff” than others, which thehuman designer of the model explicitly defines. We iterate as above, butstochastically choose among avars to adjust and amounts to adjust themso as to favor changes that preferentially move less-stiff avars andkeep avars in their preferred ranges and near their preferred centervalues. We can use an off-the-shelf constrained optimization solver forperforming this iteration quickly. Again it is rare for the knot targetto move so far in a single update pass that we need to iterate more thana couple times. This tends to happen only in highly non-linear controlgraphs. Such graphs are consequently difficult for us to manipulate viathe motion path, and users will typically resort to direct manipulationof the geometry or adjustment of the underlying avars in controllingthese structures.

When assembling a model for this system, it is worth considering howcommon non-linearities in the parameterization of articulation controlsmay affect naturalness of motion-path manipulation. For example,singularities in Euler rotations can make direct manipulation of themotion path awkward in some situations. We allow users to build modelsusing Euler angle or quaternion rotations but prefer quaternions becausethey yield more natural direct manipulation.

We provide several simple ways for users to exercise some control overthe constrained solution without modifying the underlying model or avarsetup. First, the user may lock particular avars at will, therebyeliminating corresponding degrees-of-freedom in the solution space. Themost common example of this is using an axis-constraint hotkey ormanipulation handle to limit control to a single avar. For example,rotating an Euler-angle joint by dragging on the red (+X) axisconstraint ring of the arcball will adjust only the rx avar for thatjoint. In the case of quaternion rotations, the system will project thespecified rotation onto the desired rotational axis, thereby enforcingthe axis constraint without actually causing any avars to be locked. Theuser may also explicitly lock or unlock any avar. Motion-pathmanipulation solutions will honor that lock by not considering that avarto be a degree-of-freedom. The motion path display reflects this by onlyshowing knots for “free” avars (Section 4.1.3).

The user may also control the constrained solution to some extent byadjusting the “stiffness” of the avars (as described above). We providea manipulation mode in which clicking and dragging the mouse on a jointor other node adjusts the stiffness of avars directly driving thatjoint. This is particularly useful for inverse kinematics solutions,where we prefer a linkage to bend more at one particular joint than atanother.

Finally, we may avoid the entire analytic or numerical graph inversionprocess described above by having the model explicitly define inversemappings from common manipulations (translation, rotation, scale,longitudinal bends, etc) to its avars. This is the simplest and in manyways most convenient method, but naturally it lacks flexibility forexceptional circumstances. Nonetheless, in simple models such as jointedskeletons this method may be sufficient for most or all avars.

When the user enables the option to display numerical values for theassociated avars at selected motion path knots, in addition tomanipulating a path knot itself, the user may edit the numerical labelsdirectly or may drag right and left on them to “mung” them up and down.

4.2.3 Manipulator Widgets

A model may specify supported manipulation modes and correspondingmanipulation widgets (such as the rotation arcball shown in FIG. 8 c)for use in controlling particular sets of avars. The system providesdefault manipulator widgets for translation, rotation, scale, shear,bend, squash/stretch and similar basic operations. Custom widgets may beadded as externally defined Glyph nodes through the plug-in interface(Section 1.18). All avars can be associated with some manipulation modeto be manipulated via the mechanism described in Section 4.2.2 above.Avars associated with a particular mode will be “active” only when thatmode is selected by the user.

If no other manipulation style makes sense, avars may be designated tomanipulate via “munging.” Motion path knots associated with such avarsare manipulated by mouse cursor drags in screen space; these drags mapdirectly to changes in the avar value. For example, in the case of ascalar avar, drags to the left might decrease the value of the avarwhile drags to the right would increase it. Again, it is up to themodeler to specify these mappings.

Inverse kinematics represents a special case of direct-manipulationcontrol. Here the control graph has already been inverted, and we simplyprovide direct manipulation of the IK goal. That is, to visualize motionof or control the endpoint of an IK chain, one selects the IK goal(drawn as a “jack” glyph in 3D space). The path is drawn through thegoal. The goal has simple translation, and optionally rotation, avarsthat may be manipulated through motion path knots and manipulatorwidgets as described above. Relative bend at multi-joint chains may beadjusted by modifying the relative stiffness at these joints, asdescribed above. Multiple solutions obtained by rotating about thelongitudinal axis from the IK root to the IK goal may be selected amongvia a “swing” manipulation mode; in this mode, dragging on any geometrywithin the IK chain, or on a motion path knot through any such geometry,rotates the IK chain through that solution space. More sophisticated IKchains, for example involving non-planar linkages, or non-revolutejoints, will need to be handled through the conventional graph inversionmechanism described above.

4.2.4 Other Manipulations

In addition to the ability to reposition motion path knots in 3D space,we provide several other manipulation controls. For control overrotations only, for example when the active node is a spherical joint orthe root transform of a model, we provide an axial twist control asdescribed in Section 4.1.3. This control is a manipulator handle that isdrawn when tangent handles for a selected path knot are drawn. It isdrawn from the joint-local Z-axis line, extending in a directionorthogonal to that line (e.g., +X or +Y). It indicates the axial twistaround the local Z-axis. Dragging on the handle causes the joint torotate about that axis. We determine the implied spherical quaternionrotation using an arcball-style projection of the mouse cursor onto asphere centered at the base of the twist handle. For Euler rotations, wethen extract the z-axis rotation from that spherical rotation and setonly the rz avar for that joint. For joints controlled byquaternion-valued avars, we project the spherical rotation onto thelocal z=0 plane and then convert that rz-only rotation back into aquaternion, which we use to adjusted the avar. In both cases, the avaris updated as described in Section 5.

Tangency handles at a Bezier or Hermite motion path knot at time tindicate the left and right derivatives of the path around t. The usermay click and drag with the mouse cursor on a handle to control the pathshape; we need to map this control back to the necessary changes in theunderlying avars. The desired tangent handle direction and lengthspecifies a 3D vector; we add that vector, scaled by the inverse of theframe rate, to the motion path knot position to find the desiredposition of the path target point at times t⁻=t−1/fps (for the lefthandle) or t⁺=t+1/fps (for the right handle). We pass that target pointthrough the inverse expression subgraph in exactly the same way as we dofor knot position (Sections 4.2.1 and 4.2.2). This gives us desiredvalues for each avar at t⁻ or t⁺. Taking finite differences with theavar values at t gives us an estimate of the desired derivates in eachavar, which we use directly to set the avar knot tangent handles.Sometimes it will be impossible to select a path tangent handle with themouse cursor because the tangent handle is very short and thus obscuredby the path knot itself. Clicking and dragging on the motion path knotwhile depressing a hotkey (our implementation uses the Option/Alt key)results in dragging the left knot handle. Pulling out tangent handlesfor a motion path knot with underlying linear avar knots converts thoseknots to be Beziers. Normally when dragging a knot tangent handle, onlythat handle is manipulated; the opposing tangent handle will rotateaccordingly if the handles are “mended” (not “broken”, as describedabove) but its length will not change. Dragging a knot tangent handlewhile depressing a hotkey (again we use the Option/Alt key) causes thelengths of both tangent handles to change symmetrically (regardless ofwhether the handles are “broken”): the opposite handle length “mirrors”the change in the manipulated handle. Thus, when the user “option-drags”on a path knot with null tangent handles, not only the left handle butthe right handle also are pulled out from the knot. There are menuitems, as well as a hotkey, to “break” and “mend” the tangent handles atthe selected motion path knot; these actions are applied to theunderlying avar knots. Mending broken tangent handles causes the righthandle to be set to match the slope and length of the left handle.Breaking tangent handles does not immediately change the slope or lengthof either handle, but allows the handles subsequently to be manipulatedindependently.

Motion path knots may be retimed by dragging on them while depressing ahotkey (the Control key in our implementation). Dragging to screen-leftdecreases the time of each underlying avar knot, while dragging toscreen-right increases the time of the avar knots. Knots can only beretimed in the interval between nearest preceding and succeeding pathknots. When multiple path knots are selected, retiming one path knotsimilarly retimes the others. That is, if path knots at frames 6 and 10are selected, and the knot at frame 6 is retimed to frame 7, then theknot at frame 10 will be moved to frame 11. Knot times are adjusted tothe nearest integral (whole number) frame by default, but this behaviorcan be overridden in the preferences or via a hotkey during dragging.Retiming a knot while holding an additional hotkey (Option/Alt in ourimplementation) causes the underlying avar knots to be copied to the newtime, rather than moved in time. Retiming a number of selected knotswhile holding another hotkey causes the knot times to be stretched intime (or, more precisely, dilated about the current time), causing thecorresponding motion to occur more quickly or slowly.

Depressing a hotkey (the Command key in our implementation) whileclicking the mouse cursor on a frame bead inserts a new knot (i.e., abreakdown) on all active avars at the corresponding time. We optionallyinsert the new knots as Bezier knots with tangent handles adjusted topreserve the existing shape of the path. We accomplish this bydifferentiating each avar at the frame time and solving for the tangentslope and lengths that will preserve this differential. We adjust thenew knot tangent slope and length and the proximal tangent lengths ofthe preceding and subsequent knots.

Interpolation rate along the path is adjusted in one of two ways. First,the user may drag on the frame beads displayed along the path. We solvefor the change to the nearest-knot tangent handle slope and length thatwill approximate the new frame bead position in the same way we solvefor changes to the tangent handles themselves (described above). Second,the user may adjust the interpolation rate slider on the handle itself;changes to this are mapped directly to the interpolation rate of theunderlying avar knots.

During manipulation of knot time and interpolation rate, we optionallydisplay multiple ghosts along the time interval between the precedingand subsequent motion path knots. These ghosts may be drawn as fullghosted models, as minimal skeleton “stick figures” of the full model oras skeleton stick figures of the subsection of the model articulationhierarchy being manipulated (that is, the portion of the model skeletonfrom the “active” node to the node through which the motion path isdrawn).

It should be clear in this description that the motion-path-basedmanipulation of motion timing and shape are orthogonal in our system.That is, a user manipulating the motion path may adjust the timing ofmovement along the path independently of the path shape, and vice versa.Furthermore, the motion path display and control mechanism we havedescribed is equally applicable to varied types of animated values(e.g., scalars, points, vectors, quaternions, and others) and toarbitrary expression graphs involving articulated structures, deformablesurfaces, or indeed any Turing-expressible computation. It providesreal-time direct manipulation and visualization fully decoupled fromframe rendering rates. All this is enabled and facilitated by thetemporal caching, cycle safety and related features of our expressiongraph described in Section 1.

5. Animation Layers

In Section 1.18.1, we described our use of articulated variables, oravars, to provide changing values within our expression graph. Here wepresent the heterogenous, multi-layered representation used within theseavars; their implementation; and associated mechanisms for their use. Weintroduce the idea of layers of motion, analogous to the image layersprovided in popular image-editing applications.

An avar node in our expression graph expresses some function of itsscalar input parameter. Often this parameter is time, but that need notbe the case. So far we have said little about how this function isdefined.

Avars are composed of multiple motion channels, each of which specifiessome aspect of the total avar output function. Motion channels arecomposited together under a programmable functional composition,described below. Motion channels are organized into layers. A given avarmay have zero or one motion channels for each layer. If it has zerochannels on a given layer, we say it has no animation for that layer. Ifit has one channel on a given layer, we say it has animation for thatlayer. If an avar has animation for any layer, we say that avar is set;otherwise it is unset or defaulted. If the output value of an avarvaries as its input parameter changes, we say the avar is animated;otherwise it is constant.

As described in Section 1.18.1, a given avar may have any one of anumber of value types, such as scalar, vector, point, quaternion, scalarfield, or mesh. All motion channels within that avar will have the sametype.

At a given time, a single layer is the active layer. Directmanipulation, display of knots on the motion path, and manipulation ofthe motion path all involve the active layer only, and are only possiblewhen the active layer is a kinematic layer (Section 5.2). The userchooses what layer is active and may change between active layers atwill.

5.1 Motion Compositing

There are two components to the channel composition: (1) a layer-widecompositing operator, and (2) an avar-specific layering policy. Thecompositing operator is one of the operators defined by Porter and Duffin T. Porter and T. Duff., Compositing Digital Images, Computer Graphics(Proceedings of SIGGRAPH 84), 18(3):253-259, July 1984, which article isincorporated by reference herein in its entirety. We draw the looseanalogy that our notion of “layer” corresponds to Porter and Duff's“picture” and our “motion channel” corresponds to Porter and Duff's“pixel”. Accordingly, we provide the following operators for compositingtwo layers A and B: TABLE 3 Compositing operators, after Porter and DuffOperation Name w_(A) w_(B) Interpretation (relative to A) clear clear 00 erase all animation A replace 1 0 replace all underlying animation Bignore 0 1 disable layer A A over B over 1 1 − α_(A) override specifiedavars B over A under 1 − α_(B) 1 override previously defaulted avars Ain B in α_(B) 0 keep only avars common to both layers, overridingprevious animation B in A mask 0 α_(A) mask out all but specified avarsA out B out 1 − α_(B) 0 animate only specified avars except those thatwere previously animated B out A exclude 0 1 − α_(A) mask out specifiedavars A atop B atop α_(B) 1 − α_(A) override previously animated avars Batop A mask- 1 − α_(B) α_(A) animate only specified avars, under keepany existing animation in those avars A xor B xor 1 − α_(B) 1 − α_(A)eliminate avars animated in both layers A plus B blend 1 1 blend layerstogether according to channel layering policy A plus B add 1 1 blendlayers together using an additive layering policy A plus B multiply 1 1blend layers together using a multiplicative layering policy

The operation column specifies the Porter-Duff compositing operation,name specifies the term we use to identify this operation, and w_(A) andw_(B) (F_(A) and F_(B) in Porter-Duff) are weighting factors for layersA and B as discussed below. We part ways with Porter and Duff in thatthey: (1) pre-multiply pixel values by alpha, while we do not similarlypre-scale our motion channels, and (2) in the detail that they specifyan alpha plane that provides a fractional masking value per pixel. Incontrast, our motion channels are not associated with individuallyvariable alpha values. We recall that avars lacking animation on aparticular layer do not possess a motion channel for that layer. Weconsider the alpha associated with such a non-existent channel to bezero. Channels that do exist in the layer all have the same alpha,specified layer-wide by a special envelope avar. Each layer has exactlyone envelope avar, and these avars are unique in that they arethemselves unlayered, that is, each envelope avar contains a singlemotion channel for the layer that envelope affects. The envelope avar isunaffected by the compositing operator and layer alpha. However, theenvelope avar is optionally subject to the time-course avar describedbelow. For example, the envelope for a cycling layer may optionally becycled along with the rest of the layer. As with other avars, layerenvelopes may be defaulted (to 1.0), set to a constant value, oranimated. The value-type of envelope avars is scalar, and they areusually kinematic (i.e., splined, as described below). As we shall see,setting envelopes to fractional values between 0.0 and 1.0 allowscontinuous blending between layers.

The layering policy for a given avar is either additive ormultiplicative. The policy defines how the channels of that avar combineunder compositing. For example, translation avars and most other scalaravars are additive, that is, their numerical values in layers A and Bare combined through scalar addition. Conversely, scale avars (sx,sy,sz)and unit-quaternion rotations are multiplicative: they are combined by(scalar or quaternion, respectively) multiplication. Transformationmatrices are normally multiplicative. The user who constructs theexpression graph for the model may specify either policy for a givenavar. The avar will follow the same policy on all layers except thoseusing the add or multiply operators, both of which override theavar-specific layering policy with a layer-wide policy.

During layer compositing, the layer-weighting factor (WA or WB) ismultiplied by the layer alpha (recalling that our motion channels arenot pre-multiplied by alpha), and the resulting value is used to scalethe motion channels within the layer. Scaling a motion channel isinterpreted as a linear interpolation between the channel value and atype-specific zero (for additive channels) or identity (formultiplicative channels); zero and identity are defined per typeaccording to common mathematical convention, for example, for scalarsthey are just the usual 0.0 and 1.0, respectively. Scalars, points,vectors, matrices, and most other types interpolate according to thestandard linear interpolate (lerp) operator, while unit quaternionsinterpolate via the spherical interpolate (slerp) operator. Thus for anadditive scalar avar, scaling a motion channel corresponds to a scalarmultiply of the channel value. Scaling a (multiplicative) quaternionrotation involves a slerp between quaternion identity (no rotation) andthe channel value.

For example, given two motion channels of an additive scalar avar f, theover operator would composite these avars as:f(t)=a _(A) w _(A) f _(A)(t _(A))+a _(B) w _(B) f _(B)(t _(B))=a _(A) f_(A)(t _(A))+a _(B)(1−a _(A))f _(B)(t _(B))where f_(v)(x) gives the value of avar f's motion channel for layer v atinput parameter x. For now we assume t=t_(A)=t_(B). (We will reexaminethat assumption below.) Similarly, two channels of a multiplicativequaternion avar q would composite via the atop operator as:$\begin{matrix}{{q(t)} = {\alpha_{A}w_{A}{q_{A}( t_{A} )}*\alpha_{B}w_{B}{q_{B}( t_{B} )}}} \\{= {\alpha_{A}\alpha_{B}{q_{A}( t_{A} )}*{\alpha_{B}( {1 - \alpha_{A}} )}{q_{B}( t_{B} )}}} \\{= {{{slerp}( {{\alpha_{A}\alpha_{B}},q_{identity},{q_{A}( t_{A} )}} )}{{{slerp}( {{\alpha_{B}( {1 - \alpha_{A}} )},q_{identity},{q_{B}( t_{B} )}} )}.}}}\end{matrix}$

We have assumed thus far that all motion channels are parameterizedidentically, that is, that t=t_(A)=t_(B) for all layers A and B. Now weintroduce the notion of a time-course avar. Each layer has exactly onetime-course avar and, like the envelope avar, this avar is special inthat it is unlayered, that is, each contains a single motion channel forthe layer that time-course affects. The time-course avar is unaffectedby the compositing operator, layer alpha or the layer envelope. For somelayer A, the time course avar defines a function TA mapping from avarinput parameter space to the parameterization of the evaluation functionx_(A) for motion channels within layer A. That is, t_(A)=T_(A)(t).

A particularly useful time-course function is the saw-tooth functioncurve. This causes animation within a layer to cyclically repeat(“cycle”) over some time range. Cycling is the most-common use of thetime-course avar, and, for convenience, we provide an easy mechanism inthe application GUI for cycling layers over some given range. Morecomplex time-course manipulations can be done by editing the time-courseavars directly via a spreadsheet or curve editor.

In general, layer composition for avar x involves:${x(t)} = {{\overset{0}{\coprod\limits_{i = n}}x_{i}} = {x_{n} \circ_{n}x_{n - 1} \circ_{n - 1}\cdots \circ_{2}x_{1} \circ_{1}x_{0}}}$where ∘_(i) is the (user-specified) operator for compositing layer iupon layer i-1, with associated weighting factors w_(A) and w_(B) asdefined in Table 2, and x_(i) is the value of the (existent) motionchannel for layer i. We recall that the layering policy for a givencompositing operation is the layering policy of the avar beingcomposited, except under the special compositing operators add andmultiply, as indicated in Table 2. Then:${a \circ b} = \{ {{{\begin{matrix}{a \oplus b} & {{when}\quad{layering}\quad{policy}\quad{is}\quad{additive}} \\{a \otimes b} & {{when}\quad{layering}\quad{policy}\quad{is}\quad{multiplicative}}\end{matrix}x_{i}} \oplus x_{j}} = {{{{interp}( {{w_{A}{\alpha_{i}(t)}},0,{x_{i}( {T_{i}(t)} )}} )} + {{{interp}( {{w_{B}{\alpha_{j}(t)}},0,{x_{j}( {T_{j}(t)} )}} )}{x_{i} \otimes x_{j}}}} = {{{interp}( {{w_{A}{\alpha_{i}(t)}},1,{x_{i}( {T_{i}(t)} )}} )*{{interp}( {{w_{B}{\alpha_{j}(t)}},1,{x_{j}( {T_{j}(t)} )}} )}{\alpha_{i}(t)}} = \{ \begin{matrix}{E_{i}( {T_{i}(t)} )} & {{if}\quad{time}\text{-}{course}\quad{affects}\quad{layer}\quad{i'}s\quad{envelope}} \\{E_{i}(t)} & {{otherwise}.}\end{matrix} }}} $

For quaternion-typed avars, interp is the spherical linearinterpolation, or slerp, operator; otherwise it is lerp. 0 and 1 denotetype-appropriate values for “zero” and “identity”. E_(i)(t) is the valueof the envelope avar for layer i at parameter t, and the alternativelookup strategies for E( ) reflect the user-specified option of whethera particular layer's envelope should be affected by that layer'stime-course avar.

We note that motion compositing is not mathematically commutative, thatis, the compositing order is significant, both because the Porter-Duffcompositing operators are non-commutative and because multiplication ofsome of our avar data types (e.g., matrix and quaternion tensors) isnon-commutative. However, motion compositing is mathematicallyassociative, that is, the grouping of compositing steps is notsignificant, because the Porter-Duff compositing operators areassociative and tensor multiplication is associative. In fact, onelimitation we impose on value types within our system is that thosetypes generally are associative under addition and multiplication, aswell as possess a well-defined zero and identity.

5.2 Motion Channel Taxonomy

We have said nothing yet about how the functional mapping from parametert to value x_(i) within a single motion channel is specified. In fact,rather than a single such specification, there are alternativespecifications; this choice may be made per layer. The classes of layersinclude kinematic, forward dynamic, multipoint dynamic, procedural,sampled, and stochastic. All motion channels within the same layer areof the same class.

Kinematic animation refers to animation that is defined by “splines”,that is, curves whose shape is defined sparsely via knots and aninterpolation rule for filling in the gaps between knots. Kinematicanimation is often, though not necessarily, created by a human operator.

Forward-dynamic motion refers to animation that has been generated via acomputer-simulation solution to a single-point boundary problem. Theboundary value is specified for some initial parameter, e.g., t=0, andthe simulation uses a differential equation solver to integrate forwardin time, computing values at subsequent t's. This is generally done asan offline process and the resulting animation data is saved in a tabledformat for fast access.

Multipoint-dynamic motion refers to animation that has been generatedvia a computer-simulation solution to a multi-point boundary problem.Boundary values are specified at a number of parameters t_(i), and thesimulation uses an iterative process such as gradient descent to developa solution that interpolates the specified boundaries subject to certainconstraints. Again, the solution is generally computed as an offlineprocess with the resulting animation data saved in a readily accessibleformat

Procedural animation in general is any animation computed automaticallyby a computer process. Here it refers to animation computed by a user-or modeler-supplied external function, typically written in C++ codeusing a supplied plugin API. The animation is generated by the externalfunction on an as-needed basis, though values once computed may beretained within our expression graph via temporal caching, as describedin Section 1.

Sampled motion refers to animation data recorded as discretetime-samples on disk. This includes motion capture data and the outputof externally executed computer simulations or other computer proceduresthat write their output to disk as tabled time-based samples.

Stochastic motion denotes motion computed by a pseudo-random process. Weprovide various controls for “shaping” the distribution of the randomnumber generation. In general, any user-level or code-levelrepresentation of a probability distribution function (PDF) could beused for shaping the generated motion, including the provision ofrepresentative example motion whose PDF could be extracted through anautomatic or manual process.

Any class of layer may be converted to a sampled layer by sampling themotion in that layer and outputting the time-based samples to a datafile in table format. Any class of layer may be converted to a kinematiclayer by automatically fitting spline segments to the motion in thatlayer and storing the knots that define those spline segments.

Boundary values for forward- and multipoint-dynamic simulations arespecified via selected “lower-numbered” (i.e., lower in the compositingstack) layers. These boundary-specification layers exist only to guidethe simulation and are effectively disabled when the overlying dynamiclayer is enabled.

There is no limitation on how the different classes of motion areintermixed. Layers of different classes may be composited freely oneupon another. The class for each layer is independently user-specified.

5.3 Editing Kinematic Animation

When the user wishes to modify by direct manipulation the pose of amodel, we take into account the contribution to the existing pose notonly of the active layer but of all other layers as well. We factor outthe effect of the non-active layers in deciding how to modify the activelayer. We do this by inverting the composition process around the activelayer. (For clarity, in the discussion that follows we assume that alltime-course avars give the default mappings t=T_(i)(t) and that theenvelope avar for the active layer is 1.0 everywhere. For fullgenerality, we could replace all occurrences of t with T_(i)(t) andscale the resulting f_(v)′(t) by the inverse of the envelope value, asis done in our implementation.)

That is, in order to set avar f to value y at parameter t, we computethe new value f_(v)′(t) for active (kinematic) layer v as:${f_{v}^{\prime}(t)} = \{ \begin{matrix}{( {- {\overset{0}{\coprod\limits_{i = {v - 1}}}{f_{i}(t)}}} ) \circ y \circ ( {- {\overset{n}{\coprod\limits_{i = {v + 1}}}{f_{i}(t)}}} )} & {{when}\quad{layering}\quad{policy}\quad{is}\quad{additive}} \\{( {\underset{i = {v - 1}}{\coprod\limits^{0}}{f_{i}(t)}} )^{- 1} \circ y \circ ( {\underset{i = {v + 1}}{\coprod\limits^{n}}{f_{i}(t)}} )^{- 1}} & {{when}\quad{layering}\quad{policy}\quad{is}\quad{{multiplicative}.}}\end{matrix} $

For example, suppose we wish to set an additive scalar avar f to havethe value y at parameter t by modifying only layer v. For simplicity weassume that all layers are composited under the blend operator and thatthe envelope avar for each layer is left at the default of 1.0. Then thenew value we should set in layer v is: $\begin{matrix}{{f_{v}^{\prime}(t)} = {{f_{v}(t)} + {\Delta\quad f_{v}}}} \\{= {( {- {\coprod\limits_{i = {v - 1}}^{0}{f_{i}(t)}}} ) \circ y \circ ( {- {\coprod\limits_{i = n}^{v + 1}{f_{i}(t)}}} )}} \\{= {{- ( {{f_{v - 1}(t)} + {f_{v - 2}(t)} + \cdots + {f_{0}(t)}} )} +}} \\{y + {- ( {{f_{n}(t)} + {f_{n - 1}(t)} + \cdots + {f_{v + 1}(t)}} )}} \\{= {y - {\sum\limits_{i \neq v}{f_{i}(t)}}}} \\{= {y - ( {{f(t)} - {f_{v}(t)}} )}} \\{= {y - {f(t)} + {f_{v}(t)}}} \\{= {y - {f(t)} + {f_{v}^{\prime}(t)} - {\Delta\quad f_{v}}}}\end{matrix}$ Δ  f_(v) = y − f(t).

Thus, we can see the necessary change to layer v is simply thedifference between the desired value and the current value, as expected.Taking another example, we wish to set multiplicative quaternion-valuedavar Q to value q, again at parameter t and modifying only layer v. Thenew value we should set in layer v is: $\begin{matrix}{{Q_{v}^{\prime}(t)} = {( {\coprod\limits_{i = {v - 1}}^{0}{Q_{i}(t)}} )^{- 1} \circ q \circ ( {\coprod\limits_{i = {v + 1}}^{n}{Q_{i}(t)}} )^{- 1}}} \\{= {q_{prerotation}^{- 1}\quad q\quad q_{postrotation}^{- 1}}}\end{matrix}$

where q_(prerotation) is the composite rotation due to layers prior tov, and q_(postrotation) is the composite rotation from layers subsequentto v. Naturally, the easy way to compute the pre- and post-active-layercomposite values is to use the existing compositing machinery describedin Section 5.1; we simply disable layers numbered ≧ and ≦v,respectively, and evaluate the avar at parameter t. In pseudocode:Avar::SetLayeredValue(v,t,y): EnableLayers(0..v−1);  /* disables allother layers */ prevalue = Evaluate(t); EnableLayers(v+1..n); postvalue= Evaluate(t); EnableLayers(0..n) if policy = additive then /* additivelayering policy */ newvalue := −prevalue + y + −postvalue; else /*multiplicative layering policy */ newvalue := prevalue.Inverse( ) * y *postvalue.Inverse( ); Layer(v).SetValue(t, newvalue);

Again, we may wish to scale newvalue by the inverse of the layerenvelope before actually inserting or adjusting a knot in the splinecurve for layer v.

5.4 Layer Properties

Layers may be given user-specified names, comments, and arbitrarytag-value pairs. A layer name is unique within its layer set (seebelow). Layers may be searched for and selected by name, comment ortag-value pair. A selected layer or layers may be operated upon in anumber of ways. Clearing a layer corresponds to deleting all motionchannels for that layer, leaving a layer containing no animation.Deleting a layer deletes the layer's motion channels as well as thelayer itself. The user may also create new layers; new layers initiallyhave no motion channels and thus no animation. The compositing operatorfor a layer may be changed at any time. In contrast, the layering policyfor each avar is normally set at model construction time.

Layers may be locked or unlocked. The information stored in the motionchannels of a locked layer may not be modified. Layers may be groupedinto layer groups called layer sets. All motion channels for a givenlayer set are stored on disk within a single logical file. Individuallayers may be moved between layer sets, subject to certain limitationsdescribed below; the motion channel data will correspondingly be movedbetween disk files. Layer sets may be locked or unlocked. Locking alayer set locks all layers within the set and corresponds to locking theassociated file on disk. Unlocking a layer set corresponds to unlockingthe associated file and can only be done by a given user if that userhas sufficient operating system “privileges” to unlock the file. Whenloading a layer set from disk, the contained layers will be locked ifthe disk file is locked; otherwise they will be unlocked.

Layers may be enabled or disabled. Disabling a layer is equivalent tospecifying a layer alpha of zero (See Section 5.1). Sets of layers maybe enabled and disabled together. A layer may be soloed, which isequivalent to enabling that layer and disabling all other layers. Theoutput value of an avar as it would be if the active layer were soloedis available at the avar's solo output. Changing active layers pushesdirty on all avar solo outputs.

Layers are displayed within a layer list window and optionally within alayers pane within the timeline window. The compositing order of layersmay be changed by dragging the layers into a different order in the listusing the mouse cursor. Each line of the layer list corresponds to onelayer and shows the layer name, the range if any over which the layercycles (via a sawtooth time-course function), toggles for the layer lockand enable attributes, and an indication of whether or not any of theactive avars have motion channels on that layer. There are two displaymodes for the layer list: layers may be organized by set, so that thelayer view contains a hierarchical list, or layers may be organized bycompositing order, and then the layer view gives a flat listing withlayers color-coded to indicate set membership by associating a(user-specified) color with each layer set. Next to each layer is anoptional preview tile that shows the active model (or some particularmodel specified by the user) rendered with only that layer enabled (orsome particular set of layers specified by the user). This tile animateswhen the user moves the mouse cursor over the tile, and otherwise showsa still image.

The layer pane in the timeline window additionally shows the duration intime over which particular layers (and/or layer sets, when viewedhierarchically) contain animation (that is, contain motion channels withtime-varying values) by the position of colored bars for each displayedlayer drawn under the timeline itself. The layer (or layer set) envelopemay be drawn as a curve over its corresponding layer bar, and the curvemay be directly manipulated. By setting interaction modes (withuser-interface buttons or by depressing keyboard keys) and then clickingand dragging with the mouse upon layer bars, the motion within layers orlayer sets can be retimed, scaled in time, cycled, duplicated andsimilarly manipulated. Specific layer and layer set operations arediscussed in Section 5.5—Operations on Layers.

The contribution of a particular layer to the total animation may bevisualized in several ways. The numerical data within layer motionchannels may be viewed and edited in a spreadsheet or curve editor. Theanimation may be played back while the user toggles the layer on andoff, or soloed and non-soloed, and observes the change. The animationmay be played back simultaneously in both enabled and disabled state,with geometry rendered opaquely with the layer enabled and translucently(i.e., ghosted) with the layer disabled, or vice versa. More generally,the user may specify two sets of enable states for the layers, and thenview the difference between these sets by simultaneously rendering oneset opaquely and the other set ghosted.

5.5 Operations on Layers

The user may perform various operations on the active layer, on multipleselected layers, and on layer sets. Layers may be converted to kinematicor sampled layer classes, as described in Section 5.2. Layers may belocked and unlocked, enabled and disabled, cycled (via the time-courseavar), faded (via the envelope avar), cleared and so on as described inprevious sections. Layers may be duplicated, producing a new layer withmotion channels identical to those in the original layer. Layers may berenamed. Layers may be moved into a different layer set provided thatthe user has write access to that set.

Multiple selected layers may be flattened into a single layer by mergingthe motion channels between layers. If the layers to be flattened arenot all of the same layer class, the layers will first be converted to acommon class, either kinematic (via curve fitting) or sampled (viachannel sampling) according to user preference.

A single layer may be decomposed into multiple layers in a number ofways. The motion channels affecting selected geometry may be extractedinto a new layer. The portion of all motion channels affecting aselected time range may be extracted into a new layer. More generally,the user may select some geometry and a time range, and extract themotion-channel data for the selected geometry over just the selectedtime range into a new layer. The geometry whose motion is to beextracted may be specified via regexp “regular expressions”; forexample, the user may specify all avars whose pathname matches “*Foot”,thereby identifying two avars “LeftFoot/midBend” and“RightFoot/midBend”.

The user may specify a range infrequency-space to extract into a newlayer. The channel data will be converted to the sampled class (Section5.2) and a Fast-Fourier Transform (FFT) of the channel data will beperformed. The data in the specified frequency band will then be copiedor moved (as specified in command-option dialogs) into a new layer.Several frequency ranges may be specified, producing multipleband-specific layers. By adjusting the alpha (Section 5.1) of the layersso extracted and then flattening these layers back into a single layer,the user may perform parametric equalization of the animation data. Toease this process, we provide a single-step parametric equalizationdialog with fully adjustable band centers, widths and gains for auser-specified number of bands. This is implemented by performing anFFT, decomposing by band, and recomposing with the specifiedequalization. We similarly provide a single-step notch filter using thesame technique.

We provide other analogs to audio signal processing, allowing the userto limit, compress, expand, compress-expand (compand), gate, and duckthe motion-channel signals. A compressor provides an automatic form oflevel-control, attenuating high levels and thereby reducing the dynamicrange. A limiter is an extreme form of compressor that provides verysharp attenuation above a set level. An expander attenuates the signalbelow some threshold, reducing low-level noise and expanding the dynamicrange. A gate is an extreme form of expander that sharply attenuatessignals below some threshold, suppressing noise when the signal is low.A compander combines a compressor with an expander, increasing dynamicrange while avoiding excessive output level. Ducking attenuates a signalwhen another signal exceeds some threshold, allowing one layer toautomatically override another layer when a signal (i.e., motion) ofsignificant magnitude is present, and have no effect otherwise. The usercan specify both the primary and override layers for ducking; the othereffects involve a single layer.

Channel data may be smoothed or sharpened via one-dimensionalconvolution. More generally, any finite-kernel convolution filter,specifiable by the user, may be convolved over the channels within alayer.

The difference between two layers may be extracted as a new layer.“Difference” is interpreted relative to the channel layering policy,with additive channels being subtracted and multiplicative channelsdivided.

Sampled layers may be resampled at different sampling rates. Upsamplingproduces data at a higher sample rate. Downsampling produces data at alower rate. We use a cubic resampling technique described in D. P.Mitchell and A. N. Netravali, Reconstruction filters in computergraphics. Computer Graphics, (Proceedings of SIGGRAPH 88),22(4):221-228, August 1988, which article is incorporated by referenceherein in its entirety. Other sampling methods are possible.

Entire layers may be scaled and biased. This is accomplished by applyinga gain and bias to every motion channel within the layer, in the exactlythe same way gain and bias are applied to node inputs and outputs asdescribed in Section 1.13. Similarly, a layer may be shifted orstretched in time, by retiming and stretching the animation in allchannels within the layer as described for single-channel editing inSection 4.2.4. Indeed, most of the operations that may be performed onindividual avars may also be performed across entire layers, with thesoftware simply applying the given operation to each channel within thelayer. The notable exceptions to this are operations that depend uponthe data type of an avar, for example, setting knot values.

6.0 User System Architecture

FIG. 12 is a block diagram of an exemplary user system architecture 1200for hosting an animation application. The architecture 1200 includes oneor more processors 1202 (e.g., IBM PowerPC.®, Intel Pentium® 4, etc.),one or more display devices 1204 (e.g., CRT, LCD), one or more graphicsprocessing units 1206 (e.g., NVIDIA® Quadro FX 4500, GeForce® 7800 GT,etc.), one or more network interfaces 1208 (e.g., Ethernet, FireWire,USB, etc.), one or more input devices 1210 (e.g., keyboard, mouse,etc.), and one or more computer-readable mediums 1212 (e.g. SDRAM,optical disks, hard disks, flash memory, L1 or L2 cache, etc.). Thesecomponents exchange communications and data via one or more buses 1214(e.g., EISA, PCI, PCI Express, etc.).

The term “computer-readable medium” refers to any medium thatparticipates in providing instructions to a processor 1202 forexecution, including without limitation, non-volatile media (e.g.,optical or magnetic disks), volatile media (e.g., memory) andtransmission media. Transmission media includes, without limitation,coaxial cables, copper wire and fiber optics. Transmission media canalso take the form of acoustic, light or radio frequency waves.

The computer-readable medium 1212 further includes an operating system1216 (e.g., Mac OS®, Windows®, Linux, etc.), a network communicationmodule 1218, animation files 1220, an animation application 1222 andcache memory 1236. The animation application 1222 can be integrated withother applications or be configured as a plug-in to other applications.In some implementations, the animation application 1222 includes a UImanager 1224, a rendering engine 1226, an animation engine 1228, a grapheditor 1230, expression graphs 1232 and a programming module 1234.

The operating system 1216 can be multi-user, multiprocessing,multitasking, multithreading, real-time and the like. The operatingsystem 1216 performs basic tasks, including but not limited to:recognizing input from input devices 1210; sending output to displaydevices 1204; keeping track of files and directories oncomputer-readable mediums 1212 (e.g., memory or a storage device);controlling peripheral devices (e.g., disk drives, printers, GPUs 1206,etc.); and managing traffic on the one or more buses 1214. The networkcommunications module 1218 includes various components for establishingand maintaining network connections (e.g., software for implementingcommunication protocols, such as TCP/IP, HTTP, Ethernet, etc.). Theanimation application 1220, together with its components, implements thevarious tasks and functions, as described with respect to FIGS. 1-11.The UI manager 1224 is responsible for the generation and display ofuser interfaces. The rendering and animation engines 1226, 1228, areresponsible for renderings and animating bodies in the displayenvironment. The graphic editor 1230 is described in Section 1.19. Theexpression graphs are described in Section 1. 1. The programming module1234 is responsible for the modeling, expression and animation languagesdescribed in Section 2. The cache 1236 is used in adaptive/temporalcaching described in Section 1.

The user system architecture 1100 can be implemented in any electronicor computing device capable of hosting an animation application,including but not limited to: portable or desktop computers,workstations, main frame computers, network servers, etc.

Various modifications may be made to the disclosed implementations andstill be within the scope of the following claims.

1. An animation method, comprising: providing a display environment forpresentation on a display device; providing a body for display in thedisplay environment; and determining a motion path for a first portionof the body relative to a second portion of the body.
 2. The method ofclaim 1, where determining a motion path further comprises: displaying acurve in the display environment that defines the motion path in spaceover time.
 3. The method of claim 1, further comprising: providing acontrol in the display environment for manipulating the motion path; andmanipulating the motion path in response to a user interaction with thecontrol.
 4. The method of claim 1, where the display environment is athree-dimensional space.
 5. The method of claim 1, where the displayenvironment is a two-dimensional space.
 6. The method of claim 1, wherethe body is from a group of bodies including articulated bodies, rigidbodies and bodies with one or more deformable surfaces.
 7. The method ofclaim 1, further comprising: receiving input from an input devicemanipulated by a user while viewing the body in the display environment.8. The method of claim 2, where manipulating the motion path furthercomprises: deforming one or more surfaces of the body.
 9. The method ofclaim 2, where displaying a curve further comprises: determining pointsalong the motion path at discreet sample times; and rendering the pointsto define the curve through space over time.
 10. The method of claim 9,further comprising: rendering the curve using a number of points basedat least in part on the speed of the body along the motion path.
 11. Themethod of claim 9, further comprising: automatically rendering the curveto pass through a reference point on the body in response to user input.12. The method of claim 9, further comprising: dilating the curve abouta point in space to improve visibility of the curve in the displayenvironment.
 13. The method of claim 9, further comprising: stretchingthe curve out in time to improve visibility of the curve in the displayenvironment.
 14. The method of claim 9, further comprising: renderingthe line segments relative to a moving camera view.
 15. The method ofclaim 1, further comprising: displaying one or more portions of thecurve in one or more colors to improve visibility of the curve in thedisplay environment.
 16. The method of claim 1, further comprising:displaying information along the motion path to indicate rate ofmovement of the body over a section of the motion path.
 17. The methodof claim 1, further comprising: displaying information along the motionpath to indicate timing and one or more values associated with animationkey frames.
 18. The method of claim 3, further comprising: renderingghost geometry along a section of the motion path during interactivemanipulation of the motion path.
 19. The method of claim 1, furthercomprising: animating the first portion of the body to follow the motionpath.
 20. The method of claim 19, further comprising: providing ghostimages of the first portion of the body along the motion path duringanimation.
 21. The method of claim 19, further comprising: providingcached data associated with frames of animation over time, where atleast some of the cached data is used during animation.
 22. The methodof claim 1, further comprising: defining the body by an expression graphwith cycles.
 23. The method of claim 22, further comprising: providingone or more articulated variables for changing values within theexpression graph.
 24. An animation method, comprising: providing adisplay environment for presentation on a display device; providing abody for display in the display environment; providing a first motionpath for a first portion of the body relative to a second portion of thebody; providing a second motion path for the first portion of the bodyrelative to a second portion of the body; and providing a mechanism foranimating the first portion of the body to follow either the first orsecond motion path.
 25. The method of claim 24, further comprising:associating the first motion path with a first layer; and associatingthe second motion path with a second layer.
 26. The method of claim 25,further comprising: associating the first and second layers with a classfrom a group of layer classes including: kinematic, forward dynamic,multipoint dynamic, procedural, sampled and stochastic.
 27. The methodof claim 24, further comprising: defining the first and second layers asa layer set.
 28. The method of claim 24, further comprising: performingone or more operations on the first and second layers in response touser input.
 29. A method of creating expression graphs for an animationsystem, comprising: providing a graphical user interface; receiving afirst node selection from a plurality of node types; receiving a secondnode selection from the plurality of node types; displaying graphicalrepresentations of the selected nodes in the graphical user interface;and providing a connection mechanism for enabling a user to connect atleast one output of the first node with at least one input of the secondnode to form a graphical structure that represents a unified generalizedexpression graph with cycles.
 30. The method of claim 29, where at leastsome of the node types are templates.
 31. The method of claim 29, wherethe expression includes one or more conditions.
 32. The method of claim29, further comprising: providing an expression language for enteringmathematical expressions into an input of one or more of the first andsecond nodes.
 33. The method of claim 29, where at least one of thefirst and second nodes is associated with a cache structure for storingone or more values for various discrete sample times.
 34. The method ofclaim 29, where at least one of the first and second nodes isconfigurable to broadcast a value on a named channel which can bereceived by one or more receiver nodes.
 35. The method of claim 29,where at least one of the first and second nodes is configurable tosolve kinematic problems.
 36. The method of claim 29, where a user canspecify either the first node or the second node as a root node.
 37. Themethod of claim 29, where the graphical structure is a hierarchy whichis configurable to allow data to flow bi-directionally through thehierarchy.
 38. A method of creating an expression graph for an animationsystem, comprising: providing a text-based programming language forspecifying expression graphs for an animation system; executing aprogram developed with the programming language to generate anexpression graph; and animating a body in a display environment usingthe expression graph.
 39. An animation method, comprising: providing adisplay environment for presentation on a display device; providing afirst body for display in the display environment; providing a secondbody for display in the display environment; and determining a motionpath for a portion of the first body relative to the second body.
 40. Ananimation method, comprising: providing a display environment forpresentation on a display device; providing a body for display in thedisplay environment; and determining a motion path for a portion of thebody relative to a coordinate system associated with the displayenvironment.
 41. An animation method, comprising: providing a displayenvironment for presentation on a display device; providing a body fordisplay in the display environment; evaluating an expression graphassociated with the body at a first rate; processing body geometry at asecond rate; and rendering the body in the display environment at athird rate, where the first and second rates are decoupled from thethird rate.
 42. A computer-readable medium having stored thereoninstructions which, when executed by a processor, causes the processorto perform the operations of: providing a display environment forpresentation on a display device; providing a body for display in thedisplay environment; and determining a motion path for a first portionof the body relative to a second portion of the body.
 43. Acomputer-readable medium having stored thereon instructions which, whenexecuted by a processor, causes the processor to perform the operationsof: providing a display environment for presentation on a displaydevice; providing a body for display in the display environment;providing a first motion path for a first portion of the body relativeto a second portion of the body; providing a second motion path for thefirst portion of the body relative to a second portion of the body; andproviding a mechanism for animating the first portion of the body tofollow either the first or second motion path.
 44. A computer-readablemedium having stored thereon instructions which, when executed by aprocessor, causes the processor to perform the operations of: providinga display environment for presentation on a display device; providing abody for display in the display environment; providing a first motionpath for a first portion of the body relative to a second portion of thebody; providing a second motion path for the first portion of the bodyrelative to a second portion of the body; and providing a mechanism foranimating the first portion of the body to follow either the first orsecond motion path.
 45. A computer-readable medium having stored thereoninstructions which, when executed by a processor, causes the processorto perform the operations of: providing a display environment forpresentation on a display device; providing a first body for display inthe display environment; providing a second body for display in thedisplay environment; and determining a motion path for a portion of thefirst body relative to the second body.
 46. A computer-readable mediumhaving stored thereon instructions which, when executed by a processor,causes the processor to perform the operations of: providing a displayenvironment for presentation on a display device; providing a body fordisplay in the display environment; and determining a motion path for aportion of the body relative to a coordinate system associated with thedisplay environment.
 47. A computer-readable medium having storedthereon instructions which, when executed by a processor, causes theprocessor to perform the operations of: providing a display environmentfor presentation on a display device; providing a body for display inthe display environment; evaluating an expression graph associated withthe body at a first rate; processing body geometry at a second rate; andrendering the body in the display environment at a third rate, where thefirst and second rates are decoupled from the third rate.